Re: [bmwg] Throughput definition discussions - volunteers needed Wed, 24 March 2021 19:44 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id B37F23A3439 for <>; Wed, 24 Mar 2021 12:44:16 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id ZgM6uksodL1Q for <>; Wed, 24 Mar 2021 12:44:12 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::834]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 7360B3A3436 for <>; Wed, 24 Mar 2021 12:44:12 -0700 (PDT)
Received: by with SMTP id l13so18462607qtu.9 for <>; Wed, 24 Mar 2021 12:44:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=from:to:references:in-reply-to:subject:date:message-id:mime-version :thread-index:content-language; bh=yFeMFohhGseALjHuQqNm+0u2vbC0xCV5ryj/eZzI4xo=; b=1L45atsWbFD++lxaITofgPYbdB79z4NBfPPgSMk07zMtz66ltNvEdiJtNHW7EZKZGU PwiKueVTQmkRraiITcsIoWgs4vcOUcC4jYMFx1ZXtkCtCPv3LW8Qm7gVQ/UOCp7gqHVq 5UdCA4j4qyOpDiTEefPbDAwLo6wKosd21JyZoNkeIG3WX3S8v6WPeurCv4/YPprXTr0z P2j51p+Dcn6hVB8h/4XTWVKThJQOO/7iHruDgORhe7MSbYQNirngBAhEwHpfr8y5I6fq 1bnfOa6bMzuXcidhV0KylOZCeVNS8sU0UWSg1kF7l/AhZlZHQB9MadoMm33ws3UIxquX ZmYA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:from:to:references:in-reply-to:subject:date :message-id:mime-version:thread-index:content-language; bh=yFeMFohhGseALjHuQqNm+0u2vbC0xCV5ryj/eZzI4xo=; b=QtzB8Y891dbyvU/23ExTFoLJ3XRit1E4XWgXoH1hUNLejseQiJG3C+WWO8Oil/wrDG leoXCgocj1RJ9Hk5kKhvC+3za7nHq9bbtoWCJpAOfdin6jrPCxbleJT3wuIz8NiLgr7T j5VeTgquBY62fOxH/gCpp2xfMObNIhXmoMzWT72wB6LMspV1f5qr62FjBX/hfWWmYUMe +5QfNSTJJpFZTl4Ta0kQKUZtEU8lrR5TRA6+sSIkUH6nsWEXe+P35Vd6L3UEJXg0gNWC q1u/KKRjd+fL0gGLKSgFh7MxTg6fPKyRjOmsQKWevy+z8DyOkGuDCHFEB6ouQ1/UvPM+ vmuA==
X-Gm-Message-State: AOAM532wmfQNTI9aqj98Ii3KC4onvjXPlJOCk/uoAHoI5Fcb1IEI3mrJ S3G7AVLARluFttYv9O5NAaJbVgIRmxLUSQ==
X-Google-Smtp-Source: ABdhPJwJ9jtvKkDNpQORKsVRJcNXUYMBhqGw7ArvyoDUx54NcyKcf45jWGvs7xsIfH6XSZ6Kbc3xOQ==
X-Received: by 2002:ac8:43d6:: with SMTP id w22mr4302766qtn.283.1616615050650; Wed, 24 Mar 2021 12:44:10 -0700 (PDT)
Received: from DESKTOP42TMNEU ([2601:986:8001:d660:e16b:9c24:e77b:7961]) by with ESMTPSA id a10sm2443208qkh.122.2021. (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Mar 2021 12:44:10 -0700 (PDT)
From: <>
To: "'Vratko Polak -X \(vrpolak - PANTHEON TECH SRO at Cisco\)'" <>, <>
References: <002801d7175e$36505d00$a2f11700$> <> <>
In-Reply-To: <>
Date: Wed, 24 Mar 2021 15:44:08 -0400
Message-ID: <002701d720e6$0ea8cf30$2bfa6d90$>
MIME-Version: 1.0
Content-Type: multipart/related; boundary="----=_NextPart_000_0028_01D720C4.879CD480"
X-Mailer: Microsoft Outlook 16.0
Thread-Index: AQJe4vNaP7isFd1px59ha5J2e/734AJ8IfC8AZvcwE6pY0X6MA==
Content-Language: en-us
Archived-At: <>
Subject: Re: [bmwg] Throughput definition discussions - volunteers needed
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 24 Mar 2021 19:44:17 -0000

Thanks for your input Vratko. We are taking the comment rec’d on the call and will be proposing an updated definition.  Stay tuned.




From: bmwg <> On Behalf Of Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)
Sent: Wednesday, March 24, 2021 3:42 PM
Subject: Re: [bmwg] Throughput definition discussions - volunteers needed



1. Define "load" (the one quantity test equipment varies between trials within a search).

2. Define validity criterion for trial measurement result.

3. Confirm we are searching for max valid load (not for some medium load optimum).

4. Define which quantity of the best trial ends up in the spreadsheet.

(5. I care about definitions, I do not have strong opinions about names.)


The long version:


I had no comments during the meeting,

but here is my updated understanding.


You can generalize trial [3] in various ways

and get corresponding generalizations of throughput [4]

by selecting best trial that satisfies some condition.


We are interested in "continuous traffic" trials,

where trials have configurable duration and "load" as input values.

(Other trial type has configurable "size"

and duration is the result of measurement,

but we are not interested in that trial type yet.)

Of course, test equipment has more inputs than just duration and load,

but we group them under "traffic profile" and keep constant.


A direct result of a (continuous traffic) trial is a bunch of counters,

different vendors may use different counters (so we are not standardizing those).

Indirect results are numerous, but two are important.

Validity criterion, and "forwarding rate".


For the classic throughput, "load" is the Offered Load (in pps),

validity criterion is zero loss, and forwarding rate is again Offered Load.


The general approach for finding "throughput" is to vary the load,

and select the best "forwarding rate" from result that passed validation.

Usually, valid trial at increased load implies increased forwarding rate,

so it is enough to do binary search to find max load that still gives valid result.

Other algorithms (such as MLRsearch) can be used.

I will call this approach "the search".

(An example where increased load can decrease forwarding rate

is Maximum Forwarding Rate [5], as it has no validation criterion

related to packet loss.)


I think the generalization for NGFW would look like this.

Traffic profile defines what a "transaction" is.

Test equipment has "transaction rate" as its "load" parameter.

It initiates new transactions with a constant frequency,

regardless of status of previous transactions.

Transaction rate is given in TPS (transactions per second).

Test equipment has counters that can evaluate transaction success percentage (at least after the trial).

Validation criterion is that only 0.0001% of transactions can be other than success.

Test equipment also has some L2 counters.

Forwarding rate is computed from those counters, (probably in bits per second,

not sure about inter frame gap and other details).

The search is then finding the highest transaction rate that still leads to valid trial result.

The "throughut" is then the forwarding rate at highest valid transaction rate.

Even if slightly smaller loads happen to achieve higher forwarding rate

(which does not happen, L2 load usually accelerates more rapidly than transaction rate).


Even if all of the above is true, we still need to name two quantities.

One is the "highest load still leading to valid trial results".

(In PLRsearch draft I am calling that a "critical load",

but maybe we can come up with something less generic.)

Another one is the "previously reported as throughput",

an L2-centric quantity measured at the highest valid load.

Ideally, this name would refer to the first name in some way.

Examples: "Critical Throughput" or "Forwarding Rate at Maximal Usable Load (FRaMUL)".








From: bmwg < <> > On Behalf Of Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)
Sent: Monday, 2021-March-15 15:43
To: <> 
Subject: Re: [bmwg] Throughput definition discussions - volunteers needed


> Let me know if you are interested


I am interested.


I think we already mentioned (during the meeting)

the need to distinguish allowed vs rejected traffic and packets vs bits.

RFC 2647 also mentions traffic orientation and goodput.

I think (for future proofing) we should think also about situations

when the amount of data going in differs from the amount of data coming out

(e.g encapsulation/decapsulation, encryption/decryption, compression/decompression).


> RFC 1242, 3.17 – Throughput


That is defined as a load with some additional properties.

This assumes constant load [0] in packets per second.

Stateful traffic often leads to non-constant loads,

affecting different layers differently.

I think we first need to update or replace intended load [2] (and offered load),

and base the "application throughput" on that.


Finally, I repeat my earlier comment: Counters examined during the traffic

cannot distinguish an in-flight "transaction" from a failed one.







From: bmwg < <> > On Behalf Of <> 
Sent: Friday, 2021-March-12 17:39
To: <> 
Subject: [bmwg] Throughput definition discussions - volunteers needed




The NGFW performance draft ( was reviewed in the virtual BMWG meeting yesterday. One item that proved to be a bit interesting  the discussion was around the definition of “Throughput” in section 6.3. We used what we thought was appropriate wording from RFC 2647, section 3.4 – Bit Forwarding Rate. The concern raised was specifically our use of the word Throughput. There was general consensus from the BMWG that this was confusing and for clarity sake we should use something else as the accepted BMWG definition of Throughput was defined well in RFC 1242, 3.17 – Throughput.  This discussion devolved into a circular one. It was suggested that some folks from NetSecOPEN and some folks from the BMWG get together on a call to come up with proposed wording that would be acceptable to everyone. I am looking for a few people from here to volunteer. Please let me know if you are interested.


Maciek Konstantynowicz and Al Morton have already volunteered.  The plan is to setup a call with the BMWG volunteers and a small number from NetSecOPEN to discuss. Our goal is to reach consensus on suitable wording that we can bring back to the group.


Let me know if you are interested and I will reach out to discuss timing.






Brian Monkman

Executive Director, NetSecOPEN

Office: +1-717-610-0808 

Fax: +1-717-506-0460

Mobile: +1-717-462-5422