Re: [bmwg] draft-morton-bmwg-b2b-frame-05

"MORTON, ALFRED C (AL)" <> Thu, 04 July 2019 15:47 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 40463120110 for <>; Thu, 4 Jul 2019 08:47:58 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 1fKB-B_915N9 for <>; Thu, 4 Jul 2019 08:47:55 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 5B9AB12010C for <>; Thu, 4 Jul 2019 08:47:55 -0700 (PDT)
Received: from pps.filterd ( []) by ( with SMTP id x64FjRDA010620; Thu, 4 Jul 2019 11:47:53 -0400
Received: from ( []) by with ESMTP id 2thkuf8hnm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 04 Jul 2019 11:47:53 -0400
Received: from (localhost []) by (8.14.5/8.14.5) with ESMTP id x64FlqSQ026309; Thu, 4 Jul 2019 10:47:53 -0500
Received: from ( []) by (8.14.5/8.14.5) with ESMTP id x64FlmK7026226 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 4 Jul 2019 10:47:48 -0500
Received: from ( []) by (Service) with ESMTP id 6E87840470A1; Thu, 4 Jul 2019 15:47:48 +0000 (GMT)
Received: from (unknown []) by (Service) with ESMTP id 4C51940470A0; Thu, 4 Jul 2019 15:47:48 +0000 (GMT)
Received: from (localhost []) by (8.14.5/8.14.5) with ESMTP id x64Flmfm021451; Thu, 4 Jul 2019 10:47:48 -0500
Received: from ( []) by (8.14.5/8.14.5) with ESMTP id x64FlekU021156; Thu, 4 Jul 2019 10:47:40 -0500
Received: from ( []) by (Postfix) with ESMTP id 9DB11E49C8; Thu, 4 Jul 2019 11:47:12 -0400 (EDT)
Received: from ([fe80::b09c:ff13:4487:78b6]) by ([fe80::8c6b:4b77:618f:9a01%11]) with mapi id 14.03.0439.000; Thu, 4 Jul 2019 11:46:54 -0400
From: "MORTON, ALFRED C (AL)" <>
To: "Maciek Konstantynowicz (mkonstan)" <>
CC: "" <>
Thread-Topic: draft-morton-bmwg-b2b-frame-05
Thread-Index: AQHVKoByNbM/rQa8vkyTnQVsC3xjaKa6fRyg
Date: Thu, 4 Jul 2019 15:46:53 +0000
Message-ID: <>
References: <>
In-Reply-To: <>
Accept-Language: en-US
Content-Language: en-US
x-originating-ip: []
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-07-04_07:, , signatures=0
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907040200
Archived-At: <>
Subject: Re: [bmwg] draft-morton-bmwg-b2b-frame-05
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 04 Jul 2019 15:47:58 -0000

Hi Maciek,

Thanks very much for your comments and suggestions.
Your support of this draft means much to me, because 
you have demonstrated your benchmarking expertise on 
many occasions. 

I'll respond to your comments below, and the new draft
(adopted by the working group earlier this year) 
will reflect the changes you suggested.


> -----Original Message-----
> From: Maciek Konstantynowicz (mkonstan) []
> Sent: Monday, June 24, 2019 7:32 AM
> Cc:
> Subject: Re: draft-morton-bmwg-b2b-frame-05
> Hi Al,
> I finally had time to properly review this draft.
> It is a very well written benchmarking specification that does address
> an important area of measuring ingress packet buffer capacity used for
> adapting packet arrival rate to not-always-steady packet processing rate
> capability in any packet processing system.
> It does take into consideration specifics of NFV systems (i.e.
> vswitches, other VNFs) where this capability is of elevated importance
> based on experience from OPNFV VSPERF benchmarking (backed up by a
> number of measurement references).
> The draft rightly notes that the size of this buffer matters as it
> enables storing packets disruptions in the software packet processor
> operation due to some external system interference. This a very
> important aspect of NFV and as such I would suggest to make this aspect
> more prominent in the draft, instead of burying it at the end of section
> 3, as it is a case now.
Thanks for that suggestion, this is one of the 
"learnings along the way" and it had proven to 
be a valuable motivation for this update.
There's a sentence in the introduction now,
explaining why "buffer-size matters".

> Two other general observations:
> - The goal is measuring (ingress) buffer size in front of HeaderProc,
>    per DUT definition in section 3.
>    - This works if there is no other buffering in the system under test.
>    - Suggest to add a paragraph dictating the setup where no egress
>      queue build up is possible.
I touched on this at the end of the scope section, citing 
Jacob and Lucien's RFC 8239, but I have made is a requirement

> - Proposed methodology works for setups where the DUT's (composed here
>    of "Buffer" and "HeaderProc" functions) behaviour can be measured
>    with external tester:
>    - This requires that any "noise" impacting DUT's behaviour is
>      identified and isolated.
>    - Potential sources of "noise":
>      - In-path active components, other than DUT, noted in draft as
>        "Ingress", "Egress".
>      - Operating system environment interrupting DUT operation.
>      - Shared resource(s) access collisions between DUT and some
>        off-path component(s), impacting DUT's behaviour, a.k.a. "noisy
>        neighbour" problem.
Good point, I made this a consideration among the Pre-requisites.
For example, if the links/LAN to the DUT is causing some loss,
that should always be found and fixed *first*.

>    - To deal with this e.g. for NFV DUT, the draft suggests to use
>      enhanced Binary Search with Loss Verification as specified in
>      [TST009], sec. 5.2. Plus repeating the test N times, sec. 5.3.
>      - Agree this is the right way to isolate the DUT behaviour.
Ok.... and this is where I have added references to the 
promising work-in-progress you cited below...

>    - But I am puzzled when it comes to proposed calculation of
>      "Corrected DUT Buffer Time", sec. 5.4.
>      - There "Measured Throughput" is measured as per RFC2544, instead
>        of referring to Binary Search with Loss Verification [TST009], or
>        possibly using MLRsearch [draft-vpolak-mkonstan-bmwg-mlrsearch]
>        to find NDR (non drop rate) and PDR (partial drop rate) and use
>        those to calculate the actual DUT buffer time.
We should talk about how these can apply, because we are searching
for a single loss level (zero), and need to make the changes quickly,
but I have referenced both drafts as informational in Section 5.2
(where BSxLV is introduced and described).

>      - Another potential candidate for the "Measured Throughput" is the
>        maximum measured throughput regardless of loss, metric popular
>        with academics dealing with software network processing, also
>        defined in CSIT as Maximum Receive Rate.
BMWG literature defines this as Maximum Forwarding Rate,
in RFC 2285, but this rate will be present when there is
lots of frame loss, buffers overloading everywhere in the DUT,
and the painted manufacturer's name peeling off the 
front panel due to the heat generated :-)

The latency of the "surviving packets" says something about
total buffer size, but I don't want to try to unpack that from other
sources of latency in the DUT while delivering Max Frame Rate. 

> Hope this makes sense.
[acm] It does - all very clear to me!
> Cheers,
> -Maciek