Re: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt

"Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco)" <vrpolak@cisco.com> Thu, 04 July 2019 17:42 UTC

Return-Path: <vrpolak@cisco.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 918D012014A for <bmwg@ietfa.amsl.com>; Thu, 4 Jul 2019 10:42:30 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -14.501
X-Spam-Level:
X-Spam-Status: No, score=-14.501 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=cisco.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7kJVdA4VA4Bz for <bmwg@ietfa.amsl.com>; Thu, 4 Jul 2019 10:42:27 -0700 (PDT)
Received: from alln-iport-2.cisco.com (alln-iport-2.cisco.com [173.37.142.89]) (using TLSv1.2 with cipher DHE-RSA-SEED-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 98D34120130 for <bmwg@ietf.org>; Thu, 4 Jul 2019 10:42:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=8520; q=dns/txt; s=iport; t=1562262147; x=1563471747; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=zuznWSYyoUswKF+4PcIQ2/dcm2Dx0EEk/4F0+Wp2A4U=; b=mTIIaoZvBgywjHKnlVb2bP9xX6/PJHu9sxXDBm4gt+VSiTcJGkfWYFJN lRXTch3wsxGIQF5QdIvkFc4oJTGtiuTT+/aepPeSDl50KbyQ7osx56ZOs oBdzUgXTxT5OMCSveT2KWDF9CbOkACvuVQ2aOcMK0vY6R+b4RkePhtlo5 E=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0AnAACrOR5d/4gNJK1fBxsBAQEBAwE?= =?us-ascii?q?BAQcDAQEBgVUEAQEBCwGBbCpqVDAoCoQSlG+YdBSBZwkBAQEMAQEYCwwBAYF?= =?us-ascii?q?LgnUCF4IVIzYHDgEDAQEEAQECAQVtijcBC4VKAQEBAQIBAQEhEToLBQcEAgE?= =?us-ascii?q?IEQQBAQMCJgICAiULFQgIAgQOBQiDG4F7Dw+mZoEyhDYCDkGFO4EMKAGLXhe?= =?us-ascii?q?Bf4QjPoJhAQECAQGBRQWDHYJYBIhtgzeCQoUgiECNE2wJAh2BeoZWiGuEOiO?= =?us-ascii?q?CLGyGMo4ujEFvhz+PfAIRFYEwJwkogVhwFRohgmwJgkQXgQMBAYddhT9BMYw?= =?us-ascii?q?wKQGBB4EhAQE?=
X-IronPort-AV: E=Sophos;i="5.63,451,1557187200"; d="scan'208";a="294497294"
Received: from alln-core-3.cisco.com ([173.36.13.136]) by alln-iport-2.cisco.com with ESMTP/TLS/DHE-RSA-SEED-SHA; 04 Jul 2019 17:42:26 +0000
Received: from XCH-RCD-013.cisco.com (xch-rcd-013.cisco.com [173.37.102.23]) by alln-core-3.cisco.com (8.15.2/8.15.2) with ESMTPS id x64HgPbL029567 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=FAIL); Thu, 4 Jul 2019 17:42:25 GMT
Received: from xch-rcd-014.cisco.com (173.37.102.24) by XCH-RCD-013.cisco.com (173.37.102.23) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 4 Jul 2019 12:42:24 -0500
Received: from xch-rcd-014.cisco.com ([173.37.102.24]) by XCH-RCD-014.cisco.com ([173.37.102.24]) with mapi id 15.00.1473.003; Thu, 4 Jul 2019 12:42:24 -0500
From: "Vratko Polak -X (vrpolak - PANTHEON TECHNOLOGIES at Cisco)" <vrpolak@cisco.com>
To: "acmorton@att.com" <acmorton@att.com>
CC: "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt
Thread-Index: AQHVMn/vS98wHemm50+PfvGP1/xdOaa6oEXg
Date: Thu, 4 Jul 2019 17:42:24 +0000
Message-ID: <8da32ddedf32446698dea5a982d3b8c7@XCH-RCD-014.cisco.com>
References: <156225530325.12259.9465843928150563377@ietfa.amsl.com>
In-Reply-To: <156225530325.12259.9465843928150563377@ietfa.amsl.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: [10.61.197.173]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Outbound-SMTP-Client: 173.37.102.23, xch-rcd-013.cisco.com
X-Outbound-Node: alln-core-3.cisco.com
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/FHEoeMUcY1CnvTQkWpKkO00NXA0>
Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 04 Jul 2019 17:42:32 -0000

Hi Al.

Sorry for being late with the review.
I wanted to think some things through,
and then did not find big enough chunk of time
to write everything down.

First, narrow comments.

* Second line of 1. Introduction:

> in[RFC2544], supported by the terms and definitions in [RFC1242], and

Missing space after the first "in".

* Point 4 in chapter 3 seem to be referring to two different estimates:

> It was found that the actual buffer time in the DUT could be
> estimated using results from the Throughput tests conducted
> according to Section 26.1 of [RFC2544], because it appears that
> the DUT's frame processing rate may tend to increase the estimate.

Which estimate is increased, relative to what?
Is it "implied estimate" tending to be larger than "corrected estimate"?

* Within 5.4:

> (assuming a simple model of the DUT
> composed of a buffer and a forwarding function)

A reasonable simplification, but could be mentioned in Motivation,
so 5.4 does not need a sentence in parentheses.

* Still within 5.4:

> Corrected DUT Buffer Time

This is a big one.
The correct name for this quantity could be
"DUT Buffer Time Correction".

* still 5.4:

> and the Buffer size is more accurately estimated by excluding them.

Can be turned into a formula, where
"Corrected DUT Buffer Time" can be used in a more natural meaning.

  Corrected DUT Buffer Time = Implied DUT Buffer Time - DUT Buffer Time Correction

* In 6. table:

> Min,Max,StdDev

Not sure if it is clear these refer to B2B length statistics.
Include information that these quantities are to be expressed in frames.


And now some broader comments.

* Suspended processor in production:

> useful to estimate whether frame losses will occur if
> DUT forwarding is temporarily suspended in a production deployment

This is the main benefit, and Introduction already mentions
"compensating for disruptions in the software packet processor".

But I think it would be good to classify different scenarios
leading to frame loss, that could explain both two buffer times
(implied and corrected) and how to use them outside back-to-back load.

In back-to-back test we have processor processing,
while the buffer is being filled by maximal traffic.
The time it takes to fill is the "implied buffer time".

In a hypothetic scenario where processor is suspended
and buffer is being filled by maximal traffic,
the time to fill the buffer is shorter, the "corrected buffer time".

A scenario occuring in practice has suspended processor,
but the buffer is being filled more slowly, say at throughput rate.
Deployers wishing to predict the time for the buffer to fill up
can use this formula:
  Real Buffer Time = Corrected Buffer Time * B2B Frame Rate / Real Frame Rate
That is why reporting corrected (instead of implied) buffer time is useful.

Also, it would be nice to name the scenarios and rename the buffer times.
For example, "running buffer time" and "suspended buffer time"
(instead of "implied" and "corrected" respectively).

* B2B processor rate:

For the computation of the corrected buffer time to be correct,
real processor frame processing rate (average during B2B test)
should be used instead Measured Throughput in the 5.4 formula.

The process rate is not easy to measure directly,
especially if the immediate rate varies over the duration of B2B traffic.
I agree that Throughput is a reasonable approximation,
but there may be other quantities (e.g. FRMOL [1]),
that are either a better approximation, or at least easier to measure.

Not sure how much attention other such quantities should get in the draft,
as Throughput has the advantage of avoiding some frame sizes.

* DUT vs SUT:

This is related to final items of 4. Prerequisities.

> Therefore, sources of packet loss
> that are un-related to consistent evaluation of buffer size SHOULD be
> identified and removed or mitigated.

Do we have a separate document discussing differences
between testing DUT and SUT? We should have.

Usually I prefer testing SUT (meaning no extra mitigations),
but in this case, for the analysis of the three aforementioned scenarios
to work correctly, we need to make reasonably sure
the processor is not going to get suspended during B2B test.

Also, I agree that an average result of Binary Search with Loss Verification
gives a more realistic process rate estimate
than an average result without loss verification.

Vratko.

[1] https://tools.ietf.org/html/rfc2285#page-16


-----Original Message-----
From: bmwg <bmwg-bounces@ietf.org> On Behalf Of internet-drafts@ietf.org
Sent: Thursday, 2019-July-04 17:48
To: i-d-announce@ietf.org
Cc: bmwg@ietf.org
Subject: [bmwg] I-D Action: draft-ietf-bmwg-b2b-frame-00.txt


A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Benchmarking Methodology WG of the IETF.

        Title           : Updates for the Back-to-back Frame Benchmark in RFC 2544
        Author          : Al Morton
	Filename        : draft-ietf-bmwg-b2b-frame-00.txt
	Pages           : 12
	Date            : 2019-07-04

Abstract:
   Fundamental Benchmarking Methodologies for Network Interconnect
   Devices of interest to the IETF are defined in RFC 2544.  This memo
   updates the procedures of the test to measure the Back-to-back frames
   Benchmark of RFC 2544, based on further experience.

   This memo updates Section 26.4 of RFC 2544.



The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-bmwg-b2b-frame/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-ietf-bmwg-b2b-frame-00
https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-b2b-frame-00


Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

_______________________________________________
bmwg mailing list
bmwg@ietf.org
https://www.ietf.org/mailman/listinfo/bmwg