Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05 Tue, 12 January 2021 15:45 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 57EA83A09BE for <>; Tue, 12 Jan 2021 07:45:43 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id EGmLBePJqXGW for <>; Tue, 12 Jan 2021 07:45:40 -0800 (PST)
Received: from ( [IPv6:2607:f8b0:4864:20::d2a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id CE2CA3A0983 for <>; Tue, 12 Jan 2021 07:45:40 -0800 (PST)
Received: by with SMTP id 81so4937776ioc.13 for <>; Tue, 12 Jan 2021 07:45:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=from:to:cc:references:in-reply-to:subject:date:message-id :mime-version:content-transfer-encoding:thread-index :content-language; bh=jfknqvClF7uFM6XTxfbTTwkuy0D4DTbErIo3/S3AkOM=; b=bLmjW6dt/7fScF+YkmX9LY2F0qFV7eSKuq2U0+2Uyh7b8HHi9BMtGopeP7p852PhVs CMSiSSCS/4o68EnOLzM7lUJIhVamUOyjTGErd/hr6bh5zAMM61Kk3SSI1Iy6DuzCONUA 5sNqN7hHgKwrf5UFijqkJY9ZpgF5779orbLp/VHN4zIOX8dflfJADDfD0aciBDJQVMKp dNYPf+WqK1Zhoq+BlrPW7eriiXZaWqrI12J+3xAXWmODuopuPfLjML76FqXlvsUKEnpM zfqFqHEUQk53jym8V9ul964CYYROLxEn8QIQKYpFPtoSj3jjaXCI45AM92f0TYn8wIuL /irw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:from:to:cc:references:in-reply-to:subject:date :message-id:mime-version:content-transfer-encoding:thread-index :content-language; bh=jfknqvClF7uFM6XTxfbTTwkuy0D4DTbErIo3/S3AkOM=; b=czSLPq/QlRIJZYVamjnraDKc1oswvGm0XGGV9mAodmtM0SgZoL9cz/swJGZUM/F6lc ahHUpq+xfNzB9tWCJ2PfoTy8ZxpKQ/iM8Bqs78IRNGX/rXJmIu9dS7Gh3UJNfUJzT/16 z3u/fcv5gpK2+D4vlqqLCOek4qOuuVOKjcsHDUr2C0fWYex2DmL553Z8BNifE65YJI9T d+6omP54h88cT21PxXtm2YJ6ScfYrecqjtlzzPa/4enlWUkmWboK0txGzBX7z1rzHYMb ag7LKh4NcOFMr25oO20evD/mvZlPVlsdKxPdZY7BT8PZq/hbFIGyoHwDP1HcaGKiE5Fy y7vA==
X-Gm-Message-State: AOAM5303qIJkks80eJYFXh2Fi0b8qhta6LFDF7IQieh3yBIJy+JZv5UA WMxMWeWQ/PHpYnBlzXEvSzGC5Q==
X-Google-Smtp-Source: ABdhPJwe/HydN7GYTnfLr7PHFnD/foglSW9nG+8NR8BBdDUX+apL2aX1NKByFgj2CKbreCIVKC12Iw==
X-Received: by 2002:a02:7314:: with SMTP id y20mr16933jab.78.1610466340088; Tue, 12 Jan 2021 07:45:40 -0800 (PST)
Received: from WINDOWSU6SOVGL ( []) by with ESMTPSA id e28sm1942812iov.38.2021. (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Jan 2021 07:45:39 -0800 (PST)
From: <>
To: "'MORTON, ALFRED C \(AL\)'" <>, <>
Cc: "'Bala Balarajah'" <>, "'Carsten Rossenhoevel'" <>
References: <> <> <027801d6da0e$3ac20740$b04615c0$> <018201d6e8f8$b69e2800$23da7800$> <>
In-Reply-To: <>
Date: Tue, 12 Jan 2021 10:45:38 -0500
Message-ID: <018301d6e8f9$f9ee40d0$edcac270$>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Outlook 16.0
Thread-Index: AQEuqiYZGMRTjsr9V6cISPU4hr9IhQFfZTXVAVM+aawBwIf7rAFK1qRCq0avZkA=
Content-Language: en-ca
Archived-At: <>
Subject: Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 12 Jan 2021 15:45:43 -0000

Hi Al,

We will follow your lead. 


-----Original Message-----
Sent: January 12, 2021 10:41 AM
Cc: 'Bala Balarajah' <>rg>; 'Carsten Rossenhoevel'
Subject: RE: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05

Hi Brian,

> The WGLC will close on 22 January, 2021, allow for holidays and other
competing topics.

I continue to plan to review sections that I have not reviewed in the past.

It is almost always better to keep the same version for the entire WGLC, so
that's what I suggest :-) but I'm glad to hear that you have executed many
changes to keep up with the comments!


> -----Original Message-----
> From: []
> Sent: Tuesday, January 12, 2021 10:37 AM
> Cc: 'Bala Balarajah' <>rg>; 'Carsten Rossenhoevel'
> <>
> Subject: RE: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
> Al et al,
> Given that there have been no further comments we will update the 
> draft and post it for, hopefully, final review.
> Brian
> -----Original Message-----
> From: <>
> Sent: December 24, 2020 11:03 AM
> To: 'Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)'
> <>rg>;
> Cc: 'MORTON, ALFRED C (AL)' <>om>; 'Bala Balarajah'
> <>rg>; 'Carsten Rossenhoevel' <>
> Subject: RE: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
> Vratko et al,
> See comments from the authors inline below - preceded by [authors].
> Brian
> -----Original Message-----
> From: bmwg <> On Behalf Of Vratko Polak -X 
> (vrpolak - PANTHEON TECH SRO at Cisco)
> Sent: December 23, 2020 10:31 AM
> To:
> Subject: Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
> > Please read and express your opinion on whether or not this 
> > Internet-Draft should be forwarded to the Area Directors for 
> > publication as an Informational RFC.
> The current draft is a large document, and I will have multiple comments.
> I expect some of them will be addressed by creating -06 version, so my 
> opinion is -05 should not be forwarded for publication.
> > Send your comments to this list or to co-chairs at 
> >
> The issue is, I do not have all the comments ready yet.
> In general, I need to spend some effort when turning my nebulous ideas 
> into coherent sentences (mostly because only when writing the 
> sentences I realize the topic is even more complicated than I thought 
> at first).
> Also, specifically for BMWG, I want my comments to be more complete 
> than usual.
> Not just "I do not like/understand this sentence", but give a new 
> sentence and a short explanation why the new sentence is better.
> I have two reasons for aiming for high quality comments.
> First, I imagine many people are reading this list.
> That means, if I write a lazy superficial comment, I save my time, but 
> readers will spend more time trying to reconstruct my meaning.
> (Similar to how in software development, code is written once but read 
> many
> times.) Second reason is high latency on this mailing list.
> Usually, by the time the author reacts to the comments, the reviewer 
> has switched their attention to other tasks, so it is better when the 
> first comment does not need any subsequent clarifications from the
> > allow for holidays and other competing topics
> I reserved some time before holidays, originally for improving 
> MLRsearch, but NGFW is closer to publishing so it takes precedence.
> My plan is to start with giving a few low-quality comments, mainly to 
> hint what areas I want to see improved.
> After holidays, I will write higher quality comments, one e-mail per area.
> This e-mail contains the low-quality comments (in decreasing order of 
> brevity).
> 1. Test Bed Considerations. Useful, but maybe should be expanded into 
> a separate draft.
> (Mainly expanding on "testbed reference pre-tests", and what to do if 
> they fail but we still want some results.)
> [authors]  The section "Test Bed Considerations" just gives a 
> recommendation (even though we haven't use Capital letter 
> "RECOMMEND"). The section describes the importance of the pre-test, 
> and it also gives an idea about pre-test. The Test labs or any user 
> can decide themselves, if the pre-test is needed for their test.  
> However, based on our discussions with test labs, they usually perform 
> such a pre-test. In our opinion, we should keep this section in the 
> draft. It just creates an awareness of pre-test to the readers.
> 2. Sentence with "safety margin of 10%". Unclear.
> If you want to add or subtract, name both the quantity before and 
> after the operation, so in later references it is clear which quantity 
> is referenced.
> Also, why 10% and not something else (e.g. 5%)?
> [authors] You are right. Either we need to change the wording or 
> remove the whole sentence. We suggest removing it
> 3. Is it "test bed" or "testbed"?
> I assume it means "SUT" plus "test equipment" together, but is should 
> be clarified.
> [authors] Based on Oxford and Cambridge, it should be "test bed". We 
> will solve the inconsistency  issue in the next version. A test bed 
> should also include test equipment.  we will describe this in the next
> 4. Sustain phase follows after ramp-up phase immediately, without any 
> pause, right? Then there is in-flight traffic at sustain phase start 
> and end, making it hard to get precise counters.
> [authors] We don't think we can add a pause between ramp-up and 
> sustain phase.  Since the frequency of the measurements are 2 second 
> and the total sustain phase is 300s,I don't think the in-flight 
> traffic will impact accuracy of the results.  However, we have two
suggestions here:
> 1. ask test tool vendors if there is any way to add pause between two 
> phases 2. we can describe in the draft that the measurement should 
> occur between X sec (e.g. 2sec) after ramp-up begins and X sec before 
> ramp-up ends.
> If it doesn't appear to be [possible to build in a pause we would go 
> with option 2.
> 5. Validation criteria. The draft contains terms "target throughput" 
> and "initial throughput", but also phrases like "the maximum and 
> average achievable throughput within the validation criteria".
> I am not even sure if validation criteria apply to a trial (e.g. 
> telemetry suggests test equipment behavior was not stable enough) or a 
> whole search (e.g. maximum achievable throughput is below acceptance
> [authors] Section 6 .1 describes the average throughput.  Due to the 
> behavior of stateful traffic (TCP) and also test tools behavior, 
> getting a 100% linear (stable) throughput is not easy. There will 
> always be continuous minor spikes. That's Why we chose to measure the 
> average values.
> We will remove the wording "maximum ..." in the next version. Also, we 
> will clarify that throughput means always avg. throughput. For an e.g. 
> "target throughput" means "average target throughput"
> 6. It seems the same word "throughput" is used to mean different 
> quantities depending on context.
> Close examination suggests it probably means forwarding rate [0] 
> except the offered load [1] is not given explicitly (and maybe is not 
> even constant).
> When I see "throughput" I think [2] (max offered load with no loss), 
> which does not work as generally the draft allows some loss.
> Also, some terms (e.g. "http throughput") do not refer to packets, but 
> other "transactions".
> [authors] The throughput measurement defined in [2] doesn't fit for L7 
> stateful traffic.  For example TCP retransmissions are not always 
> packet loss. Due to the test complexity and test tools behavior we 
> have to allow some transaction failures. Therefore, we needed to 
> define a different definition for the KPI throughput. Section 6.1 
> describes that the KPI measures the average Layer 2 throughput. But you
are right; the term "http
> throughput" can be considered as L7 throughput or Goodput.   We will work
> on
> this in the next draft.
> 7. SUT state affecting performance. The draft does not mention any, so 
> I think it assumes "stateless" SUT.
> An example of "stateful" SUT is NAT, where opening sessions has 
> smaller performance than forwarding on already opened sessions.
> Or maybe it is assumed any such state enters a stationary state during 
> ramp-up, so in sustain phase the performance is stable (e.g. NAT 
> sessions may be timing out, but in a stable rate).
> [authors] SUT MUST be stateful, and it must do Stateful inspection. It 
> doesn't mean that the SUT must do NAT if it is in stateful mode. NAT 
> is just another feature which can or can't be enabled and this is 
> based on the customer scenario.
> The traffic profile has limited (e.g. 10 for throughput test) 
> transactions per TCP connection and the session will be closed once 
> the transactions are completed. SUT will then remove the session 
> entries from its session table.
> This means, there will be always new stateful sessions will be opened and
> established during the sustain period as well.   Apart from this, we can
> consider whether we want to add NAT as an option feature in the 
> feature table (table 2).
> 8. Stateless or stateful traffic generation. Here stateless means 
> predetermined packets are sent at predetermined times.
> Stateful means time or content of next-to-send packet depends on time 
> or content of previously received packets.
> Draft section 7.1 looks like stateless traffic to me (think IMIX [3]), 
> while others look like stateful (you cannot count http transaction 
> rate from lossy stateless traffic).
> In general, stateful traffic is more resource intensive for test 
> equipment, so it is harder to achieve high enough offered load.
> Also, stateful traffic generation is more sensitive to packet loss and 
> latency of SUT.
> [authors] This is not IMIX [3].  IMIX [3] defines based on variable 
> packet sizes. But here in the draft, we define traffic mix based  on 
> different applications, and it's object sizes. For example an 
> application mix can be HTTPS, HTTPS, DNS (UDP), VOIP (TCP and UDP), 
> and, etc.). In this example we have a mix of stateful and stateless 
> traffic and each application has different object sizes. One object 
> can have multiple packets with different sizes. The packet sizes are 
> dependent on multiple factors namely; TCP behavior, MTU size, total 
> object size.
> Note: Stateful traffic generators MUST be used for all benchmarking 
> tests and we used/are using stateful traffic generators for the NSO 
> certification program.
> Vratko.
> [0]
> n-
> 3.6.1__;Iw!!BhdT!ycNEmzrmfiwdpp-
> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cf_9LAsk$
> [1]
> n-
> 3.5.2__;Iw!!BhdT!ycNEmzrmfiwdpp-
> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cHitzC1w$
> [2]
> n-
> 26.1__;Iw!!BhdT!ycNEmzrmfiwdpp-
> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cm06wZVM$
> [3]
> dT!y cNEmzrmfiwdpp-9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cDmT8_Wg$
> -----Original Message-----
> From: bmwg <> On Behalf Of MORTON, ALFRED C (AL)
> Sent: Friday, 2020-December-18 19:16
> To:
> Subject: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
> Hi BMWG,
> We will start a WG Last Call for
> Benchmarking Methodology for Network Security Device Performance
> g-
> ngfw-performance-05__;!!BhdT!ycNEmzrmfiwdpp-
> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cIm1m1L8$
> The WGLC will close on 22 January, 2021, allow for holidays and other 
> competing topics (IOW, plenty of time!)
> Please read and express your opinion on whether or not this 
> Internet-Draft should be forwarded to the Area Directors for 
> publication as an Informational RFC.  Send your comments to this list 
> or to co-chairs at
> for the co-chairs,
> Al
> _______________________________________________
> bmwg mailing list
> !BhdT!ycNEmzrmfiwdpp-9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cuBcgfP4$
> _______________________________________________
> bmwg mailing list
> !BhdT!ycNEmzrmfiwdpp-9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cuBcgfP4$