Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05

["MORTON, ALFRED C (AL)" <[e-mail]>]

Hi Authors and Contributors,

IMO, This memo is in great shape, despite my many comments at this stage.
There are some BMWG-consistency items to take care of when it comes to
address-space and metrics. Since I started my review independent of 
Vratko's comments, I continued on that way. Some of my comments may
already be resolved or overtaken by events, so to say.

As a working group, we need to decide the "Update" status of this memo.

 - We could indicate that this memo *Obsoletes* RFC 3511 methods, if that is 
the consensus. I'm fairly sure we need to say something definitive w.r.t.
RFC 3511, in IETF terms.
 - I'm fairly convinced that RFC2647 is still useful and probably needs to be 
referenced in section 6. Again, what does the WG say?

I "marked-up" the TXT version of the draft, attached, just search for "@@@@ acm"
comments.

Al
(as a participant)

>-----Original Message-----
>From: bmwg [mailto:[e-mail]] On Behalf Of MORTON, ALFRED C
>(AL)
>Sent: Tuesday, January 12, 2021 10:41 AM
>To: [e-mail]; [e-mail]
>Cc: 'Bala Balarajah' <[e-mail]>
>Subject: Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
>
>*** Security Advisory: This Message Originated Outside of AT&T ***.
>Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
>
>Hi Brian,
>
>> The WGLC will close on 22 January, 2021, allow for holidays and other
>competing topics.
>
>I continue to plan to review sections that I have not reviewed in the past.
>
>It is almost always better to keep the same version for the entire WGLC, so
>that's what I suggest :-) but I'm glad to hear that you have executed many
>changes to keep up with the comments!
>
>Al
>
>
>> -----Original Message-----
>> From: [e-mail] [mailto:[e-mail]]
>> Sent: Tuesday, January 12, 2021 10:37 AM
>> To: [e-mail]; MORTON, ALFRED C (AL) <[e-mail]>
>> Cc: 'Bala Balarajah' <[e-mail]>; 'Carsten Rossenhoevel'
>> <[e-mail]>
>> Subject: RE: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
>>
>> Al et al,
>>
>> Given that there have been no further comments we will update the draft
>> and
>> post it for, hopefully, final review.
>>
>> Brian
>>
>> -----Original Message-----
>> From: [e-mail] <[e-mail]>
>> Sent: December 24, 2020 11:03 AM
>> To: 'Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)'
>> <vrpolak=[e-mail]>; [e-mail]
>> Cc: 'MORTON, ALFRED C (AL)' <[e-mail]>; 'Bala Balarajah'
>> <[e-mail]>; 'Carsten Rossenhoevel' <[e-mail]>
>> Subject: RE: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
>>
>> Vratko et al,
>>
>> See comments from the authors inline below - preceded by [authors].
>>
>> Brian
>>
>> -----Original Message-----
>> From: bmwg <[e-mail]> On Behalf Of Vratko Polak -X (vrpolak
>-
>> PANTHEON TECH SRO at Cisco)
>> Sent: December 23, 2020 10:31 AM
>> To: [e-mail]
>> Cc: MORTON, ALFRED C (AL) <[e-mail]>
>> Subject: Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
>>
>> > Please read and express your opinion on whether or not this
>> > Internet-Draft should be forwarded to the Area Directors for
>> > publication as an Informational RFC.
>>
>> The current draft is a large document, and I will have multiple comments.
>> I expect some of them will be addressed by creating -06 version, so my
>> opinion is -05 should not be forwarded for publication.
>>
>> > Send your comments to this list or to co-chairs at
>> > [e-mail]
>>
>> The issue is, I do not have all the comments ready yet.
>> In general, I need to spend some effort when turning my nebulous ideas
>> into
>> coherent sentences (mostly because only when writing the sentences I
>> realize
>> the topic is even more complicated than I thought at first).
>>
>> Also, specifically for BMWG, I want my comments to be more complete than
>> usual.
>> Not just "I do not like/understand this sentence", but give a new
>sentence
>> and a short explanation why the new sentence is better.
>> I have two reasons for aiming for high quality comments.
>> First, I imagine many people are reading this list.
>> That means, if I write a lazy superficial comment, I save my time, but
>> readers will spend more time trying to reconstruct my meaning.
>> (Similar to how in software development, code is written once but read
>> many
>> times.) Second reason is high latency on this mailing list.
>> Usually, by the time the author reacts to the comments, the reviewer has
>> switched their attention to other tasks, so it is better when the first
>> comment does not need any subsequent clarifications from the reviewer.
>>
>> > allow for holidays and other competing topics
>>
>> I reserved some time before holidays, originally for improving MLRsearch,
>> but NGFW is closer to publishing so it takes precedence.
>>
>> My plan is to start with giving a few low-quality comments, mainly to
>hint
>> what areas I want to see improved.
>> After holidays, I will write higher quality comments, one e-mail per
>area.
>> This e-mail contains the low-quality comments (in decreasing order of
>> brevity).
>>
>> 1. Test Bed Considerations. Useful, but maybe should be expanded into a
>> separate draft.
>> (Mainly expanding on "testbed reference pre-tests", and what to do if
>they
>> fail but we still want some results.)
>>
>> [authors]  The section "Test Bed Considerations" just gives a
>> recommendation
>> (even though we haven't use Capital letter "RECOMMEND"). The section
>> describes the importance of the pre-test, and it also gives an idea about
>> pre-test. The Test labs or any user can decide themselves, if the pre-
>test
>> is needed for their test.  However, based on our discussions with test
>> labs,
>> they usually perform such a pre-test. In our opinion, we should keep this
>> section in the draft. It just creates an awareness of pre-test to the
>> readers.
>>
>> 2. Sentence with "safety margin of 10%". Unclear.
>> If you want to add or subtract, name both the quantity before and after
>> the
>> operation, so in later references it is clear which quantity is
>> referenced.
>> Also, why 10% and not something else (e.g. 5%)?
>>
>> [authors] You are right. Either we need to change the wording or remove
>> the
>> whole sentence. We suggest removing it
>>
>> 3. Is it "test bed" or "testbed"?
>> I assume it means "SUT" plus "test equipment" together, but is should be
>> clarified.
>>
>> [authors] Based on Oxford and Cambridge, it should be "test bed". We will
>> solve the inconsistency  issue in the next version. A test bed should
>also
>> include test equipment.  we will describe this in the next version.
>>
>> 4. Sustain phase follows after ramp-up phase immediately, without any
>> pause,
>> right? Then there is in-flight traffic at sustain phase start and end,
>> making it hard to get precise counters.
>>
>> [authors] We don't think we can add a pause between ramp-up and sustain
>> phase.  Since the frequency of the measurements are 2 second and the
>total
>> sustain phase is 300s,I don't think the in-flight traffic will impact
>> accuracy of the results.  However, we have two suggestions here:
>> 1. ask test tool vendors if there is any way to add pause between two
>> phases
>> 2. we can describe in the draft that the measurement should occur between
>> X
>> sec (e.g. 2sec) after ramp-up begins and X sec before ramp-up ends.
>> If it doesn't appear to be [possible to build in a pause we would go with
>> option 2.
>>
>> 5. Validation criteria. The draft contains terms "target throughput" and
>> "initial throughput", but also phrases like "the maximum and average
>> achievable throughput within the validation criteria".
>> I am not even sure if validation criteria apply to a trial (e.g.
>telemetry
>> suggests test equipment behavior was not stable enough) or a whole search
>> (e.g. maximum achievable throughput is below acceptance threshold).
>>
>> [authors] Section 6 .1 describes the average throughput.  Due to the
>> behavior of stateful traffic (TCP) and also test tools behavior, getting
>a
>> 100% linear (stable) throughput is not easy. There will always be
>> continuous
>> minor spikes. That's Why we chose to measure the average values.
>> We will remove the wording "maximum ..." in the next version. Also, we
>> will
>> clarify that throughput means always avg. throughput. For an e.g. "target
>> throughput" means "average target throughput"
>>
>> 6. It seems the same word "throughput" is used to mean different
>> quantities
>> depending on context.
>> Close examination suggests it probably means forwarding rate [0] except
>> the
>> offered load [1] is not given explicitly (and maybe is not even
>constant).
>> When I see "throughput" I think [2] (max offered load with no loss),
>which
>> does not work as generally the draft allows some loss.
>> Also, some terms (e.g. "http throughput") do not refer to packets, but
>> other
>> "transactions".
>>
>> [authors] The throughput measurement defined in [2] doesn't fit for L7
>> stateful traffic.  For example TCP retransmissions are not always packet
>> loss. Due to the test complexity and test tools behavior we have to allow
>> some transaction failures. Therefore, we needed to define a different
>> definition for the KPI throughput. Section 6.1 describes that the KPI
>> measures the average Layer 2 throughput. But you are right; the term
>"http
>> throughput" can be considered as L7 throughput or Goodput.   We will work
>> on
>> this in the next draft.
>>
>> 7. SUT state affecting performance. The draft does not mention any, so I
>> think it assumes "stateless" SUT.
>> An example of "stateful" SUT is NAT, where opening sessions has smaller
>> performance than forwarding on already opened sessions.
>> Or maybe it is assumed any such state enters a stationary state during
>> ramp-up, so in sustain phase the performance is stable (e.g. NAT sessions
>> may be timing out, but in a stable rate).
>>
>> [authors] SUT MUST be stateful, and it must do Stateful inspection. It
>> doesn't mean that the SUT must do NAT if it is in stateful mode. NAT is
>> just
>> another feature which can or can't be enabled and this is based on the
>> customer scenario.
>> The traffic profile has limited (e.g. 10 for throughput test)
>transactions
>> per TCP connection and the session will be closed once the transactions
>> are
>> completed. SUT will then remove the session entries from its session
>> table.
>> This means, there will be always new stateful sessions will be opened and
>> established during the sustain period as well.   Apart from this, we can
>> consider whether we want to add NAT as an option feature in the feature
>> table (table 2).
>>
>> 8. Stateless or stateful traffic generation. Here stateless means
>> predetermined packets are sent at predetermined times.
>> Stateful means time or content of next-to-send packet depends on time or
>> content of previously received packets.
>> Draft section 7.1 looks like stateless traffic to me (think IMIX [3]),
>> while
>> others look like stateful (you cannot count http transaction rate from
>> lossy
>> stateless traffic).
>> In general, stateful traffic is more resource intensive for test
>> equipment,
>> so it is harder to achieve high enough offered load.
>> Also, stateful traffic generation is more sensitive to packet loss and
>> latency of SUT.
>>
>> [authors] This is not IMIX [3].  IMIX [3] defines based on variable
>packet
>> sizes. But here in the draft, we define traffic mix based  on different
>> applications, and it's object sizes. For example an application mix can
>be
>> HTTPS, HTTPS, DNS (UDP), VOIP (TCP and UDP), and, etc.). In this example
>> we
>> have a mix of stateful and stateless traffic and each application has
>> different object sizes. One object can have multiple packets with
>> different
>> sizes. The packet sizes are dependent on multiple factors namely; TCP
>> behavior, MTU size, total object size.
>> Note: Stateful traffic generators MUST be used for all benchmarking tests
>> and we used/are using stateful traffic generators for the NSO
>> certification
>> program.
>>
>> Vratko.
>>
>> [0]
>> https://urldefense.com/v3/__https://tools.ietf.org/html/rfc2285*section-
>> 3.6.1__;Iw!!BhdT!ycNEmzrmfiwdpp-
>> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cf_9LAsk$
>> [1]
>> https://urldefense.com/v3/__https://tools.ietf.org/html/rfc2285*section-
>> 3.5.2__;Iw!!BhdT!ycNEmzrmfiwdpp-
>> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cHitzC1w$
>> [2]
>> https://urldefense.com/v3/__https://tools.ietf.org/html/rfc2544*section-
>> 26.1__;Iw!!BhdT!ycNEmzrmfiwdpp-
>> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cm06wZVM$
>> [3]
>>
>https://urldefense.com/v3/__https://tools.ietf.org/html/rfc6985__;!!BhdT!y
>> cNEmzrmfiwdpp-9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cDmT8_Wg$
>>
>> -----Original Message-----
>> From: bmwg <[e-mail]> On Behalf Of MORTON, ALFRED C (AL)
>> Sent: Friday, 2020-December-18 19:16
>> To: [e-mail]
>> Subject: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
>>
>> Hi BMWG,
>>
>> We will start a WG Last Call for
>>
>> Benchmarking Methodology for Network Security Device Performance
>>
>> https://urldefense.com/v3/__https://tools.ietf.org/html/draft-ietf-bmwg-
>> ngfw-performance-05__;!!BhdT!ycNEmzrmfiwdpp-
>> 9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cIm1m1L8$
>>
>> The WGLC will close on 22 January, 2021, allow for holidays and other
>> competing topics (IOW, plenty of time!)
>>
>> Please read and express your opinion on whether or not this Internet-
>Draft
>> should be forwarded to the Area Directors for publication as an
>> Informational RFC.  Send your comments to this list or to co-chairs at
>> [e-mail]
>>
>> for the co-chairs,
>> Al
>>
>> _______________________________________________
>> bmwg mailing list
>> [e-mail]
>>
>https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!
>> !BhdT!ycNEmzrmfiwdpp-9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cuBcgfP4$
>> _______________________________________________
>> bmwg mailing list
>> [e-mail]
>>
>https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!
>> !BhdT!ycNEmzrmfiwdpp-9vduF8oGkZmLCdHFdWLPW7AXW6BOzFW4leuw1EQ6cuBcgfP4$
>>
>
>_______________________________________________
>bmwg mailing list
>[e-mail]
>https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/bmwg__;!!
>BhdT!15trY7CQTBh7uiAMcRe10nfzqSEeelu_CGN174Odz_lLXeAa6E5JrIT3DZ5X$