Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05

["Maciek Konstantynowicz (mkonstan)" <[e-mail]>]

Hi Brian et al,

Thank you for producing this draft.

It is a very well structured and well written benchmarking spec
addressing a very complex and fast evolving networking security domain.
The best write-up I have ever seen. Kudos for doing the work!

I have read the draft in the past, and have been planning a more
complete detailed review for some time, but work life takes its twists
and turns, as we all know. It was the WGLC that made me finally
prioritise this work.

However, I have underestimated the amount of work required (it is quite
a large spec, 1439 non-empty lines), and I clearly failed to complete
the review before the deadline.

But what I completed, I attach. Hopefully it's of value :)

Cheers,
Maciek



> On 12 Jan 2021, at 15:41, MORTON, ALFRED C (AL) <[e-mail]> wrote:
>
> Hi Brian,
>
>> The WGLC will close on 22 January, 2021, allow for holidays and other
> competing topics.
>
> I continue to plan to review sections that I have not reviewed in the past.
>
> It is almost always better to keep the same version for the entire WGLC, so that's what I suggest :-) but I'm glad to hear that you have executed many changes to keep up with the comments!
>
> Al
>
>
>> -----Original Message-----
>> From: [e-mail] [mailto:[e-mail]]
>> Sent: Tuesday, January 12, 2021 10:37 AM
>> To: [e-mail]; MORTON, ALFRED C (AL) <[e-mail]>
>> Cc: 'Bala Balarajah' <[e-mail]>; 'Carsten Rossenhoevel'
>> <[e-mail]>
>> Subject: RE: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
>>
>> Al et al,
>>
>> Given that there have been no further comments we will update the draft
>> and
>> post it for, hopefully, final review.
>>
>> Brian
>>
>> -----Original Message-----
>> From: [e-mail] <[e-mail]>
>> Sent: December 24, 2020 11:03 AM
>> To: 'Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)'
>> <vrpolak=[e-mail]>; [e-mail]
>> Cc: 'MORTON, ALFRED C (AL)' <[e-mail]>; 'Bala Balarajah'
>> <[e-mail]>; 'Carsten Rossenhoevel' <[e-mail]>
>> Subject: RE: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
>>
>> Vratko et al,
>>
>> See comments from the authors inline below - preceded by [authors].
>>
>> Brian
>>
>> -----Original Message-----
>> From: bmwg <[e-mail]> On Behalf Of Vratko Polak -X (vrpolak -
>> PANTHEON TECH SRO at Cisco)
>> Sent: December 23, 2020 10:31 AM
>> To: [e-mail]
>> Cc: MORTON, ALFRED C (AL) <[e-mail]>
>> Subject: Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
>>
>>> Please read and express your opinion on whether or not this
>>> Internet-Draft should be forwarded to the Area Directors for
>>> publication as an Informational RFC.
>>
>> The current draft is a large document, and I will have multiple comments.
>> I expect some of them will be addressed by creating -06 version, so my
>> opinion is -05 should not be forwarded for publication.
>>
>>> Send your comments to this list or to co-chairs at
>>> [e-mail]
>>
>> The issue is, I do not have all the comments ready yet.
>> In general, I need to spend some effort when turning my nebulous ideas
>> into
>> coherent sentences (mostly because only when writing the sentences I
>> realize
>> the topic is even more complicated than I thought at first).
>>
>> Also, specifically for BMWG, I want my comments to be more complete than
>> usual.
>> Not just "I do not like/understand this sentence", but give a new sentence
>> and a short explanation why the new sentence is better.
>> I have two reasons for aiming for high quality comments.
>> First, I imagine many people are reading this list.
>> That means, if I write a lazy superficial comment, I save my time, but
>> readers will spend more time trying to reconstruct my meaning.
>> (Similar to how in software development, code is written once but read
>> many
>> times.) Second reason is high latency on this mailing list.
>> Usually, by the time the author reacts to the comments, the reviewer has
>> switched their attention to other tasks, so it is better when the first
>> comment does not need any subsequent clarifications from the reviewer.
>>
>>> allow for holidays and other competing topics
>>
>> I reserved some time before holidays, originally for improving MLRsearch,
>> but NGFW is closer to publishing so it takes precedence.
>>
>> My plan is to start with giving a few low-quality comments, mainly to hint
>> what areas I want to see improved.
>> After holidays, I will write higher quality comments, one e-mail per area.
>> This e-mail contains the low-quality comments (in decreasing order of
>> brevity).
>>
>> 1. Test Bed Considerations. Useful, but maybe should be expanded into a
>> separate draft.
>> (Mainly expanding on "testbed reference pre-tests", and what to do if they
>> fail but we still want some results.)
>>
>> [authors]  The section "Test Bed Considerations" just gives a
>> recommendation
>> (even though we haven't use Capital letter "RECOMMEND"). The section
>> describes the importance of the pre-test, and it also gives an idea about
>> pre-test. The Test labs or any user can decide themselves, if the pre-test
>> is needed for their test.  However, based on our discussions with test
>> labs,
>> they usually perform such a pre-test. In our opinion, we should keep this
>> section in the draft. It just creates an awareness of pre-test to the
>> readers.
>>
>> 2. Sentence with "safety margin of 10%". Unclear.
>> If you want to add or subtract, name both the quantity before and after
>> the
>> operation, so in later references it is clear which quantity is
>> referenced.
>> Also, why 10% and not something else (e.g. 5%)?
>>
>> [authors] You are right. Either we need to change the wording or remove
>> the
>> whole sentence. We suggest removing it
>>
>> 3. Is it "test bed" or "testbed"?
>> I assume it means "SUT" plus "test equipment" together, but is should be
>> clarified.
>>
>> [authors] Based on Oxford and Cambridge, it should be "test bed". We will
>> solve the inconsistency  issue in the next version. A test bed should also
>> include test equipment.  we will describe this in the next version.
>>
>> 4. Sustain phase follows after ramp-up phase immediately, without any
>> pause,
>> right? Then there is in-flight traffic at sustain phase start and end,
>> making it hard to get precise counters.
>>
>> [authors] We don't think we can add a pause between ramp-up and sustain
>> phase.  Since the frequency of the measurements are 2 second and the total
>> sustain phase is 300s,I don't think the in-flight traffic will impact
>> accuracy of the results.  However, we have two suggestions here:
>> 1. ask test tool vendors if there is any way to add pause between two
>> phases
>> 2. we can describe in the draft that the measurement should occur between
>> X
>> sec (e.g. 2sec) after ramp-up begins and X sec before ramp-up ends.
>> If it doesn't appear to be [possible to build in a pause we would go with
>> option 2.
>>
>> 5. Validation criteria. The draft contains terms "target throughput" and
>> "initial throughput", but also phrases like "the maximum and average
>> achievable throughput within the validation criteria".
>> I am not even sure if validation criteria apply to a trial (e.g. telemetry
>> suggests test equipment behavior was not stable enough) or a whole search
>> (e.g. maximum achievable throughput is below acceptance threshold).
>>
>> [authors] Section 6 .1 describes the average throughput.  Due to the
>> behavior of stateful traffic (TCP) and also test tools behavior, getting a
>> 100% linear (stable) throughput is not easy. There will always be
>> continuous
>> minor spikes. That's Why we chose to measure the average values.
>> We will remove the wording "maximum ..." in the next version. Also, we
>> will
>> clarify that throughput means always avg. throughput. For an e.g. "target
>> throughput" means "average target throughput"
>>
>> 6. It seems the same word "throughput" is used to mean different
>> quantities
>> depending on context.
>> Close examination suggests it probably means forwarding rate [0] except
>> the
>> offered load [1] is not given explicitly (and maybe is not even constant).
>> When I see "throughput" I think [2] (max offered load with no loss), which
>> does not work as generally the draft allows some loss.
>> Also, some terms (e.g. "http throughput") do not refer to packets, but
>> other
>> "transactions".
>>
>> [authors] The throughput measurement defined in [2] doesn't fit for L7
>> stateful traffic.  For example TCP retransmissions are not always packet
>> loss. Due to the test complexity and test tools behavior we have to allow
>> some transaction failures. Therefore, we needed to define a different
>> definition for the KPI throughput. Section 6.1 describes that the KPI
>> measures the average Layer 2 throughput. But you are right; the term "http
>> throughput" can be considered as L7 throughput or Goodput.   We will work
>> on
>> this in the next draft.
>>
>> 7. SUT state affecting performance. The draft does not mention any, so I
>> think it assumes "stateless" SUT.
>> An example of "stateful" SUT is NAT, where opening sessions has smaller
>> performance than forwarding on already opened sessions.
>> Or maybe it is assumed any such state enters a stationary state during
>> ramp-up, so in sustain phase the performance is stable (e.g. NAT sessions
>> may be timing out, but in a stable rate).
>>
>> [authors] SUT MUST be stateful, and it must do Stateful inspection. It
>> doesn't mean that the SUT must do NAT if it is in stateful mode. NAT is
>> just
>> another feature which can or can't be enabled and this is based on the
>> customer scenario.
>> The traffic profile has limited (e.g. 10 for throughput test) transactions
>> per TCP connection and the session will be closed once the transactions
>> are
>> completed. SUT will then remove the session entries from its session
>> table.
>> This means, there will be always new stateful sessions will be opened and
>> established during the sustain period as well.   Apart from this, we can
>> consider whether we want to add NAT as an option feature in the feature
>> table (table 2).
>>
>> 8. Stateless or stateful traffic generation. Here stateless means
>> predetermined packets are sent at predetermined times.
>> Stateful means time or content of next-to-send packet depends on time or
>> content of previously received packets.
>> Draft section 7.1 looks like stateless traffic to me (think IMIX [3]),
>> while
>> others look like stateful (you cannot count http transaction rate from
>> lossy
>> stateless traffic).
>> In general, stateful traffic is more resource intensive for test
>> equipment,
>> so it is harder to achieve high enough offered load.
>> Also, stateful traffic generation is more sensitive to packet loss and
>> latency of SUT.
>>
>> [authors] This is not IMIX [3].  IMIX [3] defines based on variable packet
>> sizes. But here in the draft, we define traffic mix based  on different
>> applications, and it's object sizes. For example an application mix can be
>> HTTPS, HTTPS, DNS (UDP), VOIP (TCP and UDP), and, etc.). In this example
>> we
>> have a mix of stateful and stateless traffic and each application has
>> different object sizes. One object can have multiple packets with
>> different
>> sizes. The packet sizes are dependent on multiple factors namely; TCP
>> behavior, MTU size, total object size.
>> Note: Stateful traffic generators MUST be used for all benchmarking tests
>> and we used/are using stateful traffic generators for the NSO
>> certification
>> program.
>>
>> Vratko.
>>
>> [0]
>> https://urldefense.com/v3/__https://tools.ietf.org/html/rfc2285*section-
>> 3.6.1__;Iw!!BhdT!ycNEmzrmfiwdpp-
>> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cf_9LAsk$
>> [1]
>> https://urldefense.com/v3/__https://tools.ietf.org/html/rfc2285*section-
>> 3.5.2__;Iw!!BhdT!ycNEmzrmfiwdpp-
>> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cHitzC1w$
>> [2]
>> https://urldefense.com/v3/__https://tools.ietf.org/html/rfc2544*section-
>> 26.1__;Iw!!BhdT!ycNEmzrmfiwdpp-
>> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cm06wZVM$
>> [3]
>> https://urldefense.com/v3/__https://tools.ietf.org/html/rfc6985__;!!BhdT!y
>> cNEmzrmfiwdpp-9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cDmT8_Wg$
>>
>> -----Original Message-----
>> From: bmwg <[e-mail]> On Behalf Of MORTON, ALFRED C (AL)
>> Sent: Friday, 2020-December-18 19:16
>> To: [e-mail]
>> Subject: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
>>
>> Hi BMWG,
>>
>> We will start a WG Last Call for
>>
>> Benchmarking Methodology for Network Security Device Performance
>>
>> https://urldefense.com/v3/__https://tools.ietf.org/html/draft-ietf-bmwg-
>> ngfw-performance-05__;!!BhdT!ycNEmzrmfiwdpp-
>> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cIm1m1L8$
>>
>> The WGLC will close on 22 January, 2021, allow for holidays and other
>> competing topics (IOW, plenty of time!)
>>
>> Please read and express your opinion on whether or not this Internet-Draft
>> should be forwarded to the Area Directors for publication as an
>> Informational RFC.  Send your comments to this list or to co-chairs at
>> [e-mail]
>>
>> for the co-chairs,
>> Al
>>
>> _______________________________________________
>> bmwg mailing list
>> [e-mail]
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!
>> !BhdT!ycNEmzrmfiwdpp-9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cuBcgfP4$
>> _______________________________________________
>> bmwg mailing list
>> [e-mail]
>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!
>> !BhdT!ycNEmzrmfiwdpp-9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cuBcgfP4$
>>
>
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/bmwg