[bmwg] Comments on draft-ietf-bmwg-ngfw-performance-00

["MORTON, ALFRED C (AL)" <[e-mail]>]

Hi draft-ietf-bmwg-ngfw-performance authors,

Thanks for preparing a very complete and detailed
draft!

I have the following comments (through section 7.5,
there is some repetition for HTTPS after that), 
which I hope will help.

regards,
Al
(as a participant)

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

3.  Scope

   This document provides testing terminology and testing methodology
   for next-generation firewalls and related security functions.  It
   covers two main areas: Performance benchmarks and security
   effectiveness testing.  This document focuses on advanced, realistic,
[acm] can you define "security effectiveness testing" here,
to distinguish it from performance benchmarking?


4.2.  DUT/SUT Configuration

   A unique DUT/SUT configuration MUST be used for all benchmarking
   tests described in Section 7.  Since each DUT/SUT will have their own
   unique configuration, users SHOULD configure their device with the
   same parameters that would be used in the actual deployment of the
   device or a typical deployment. * Users MUST enable security features *
   on the DUT/SUT to achieve maximum security coverage for a specific
   deployment scenario.
[acm] Is a compliant response to this requirement described below?
In Table 1?

   This document attempts to define the recommended security features
   which SHOULD be consistently enabled for all the benchmarking tests
   described in Section 7.  Table 1 below describes the RECOMMENDED sets
   of feature list which SHOULD be configured on the DUT/SUT.
[acm] SHOULD does not match the term Mandatory in Table 1.
maybe s/Mandatory/RECOMMENDED/
and s/Optional/OPTIONAL/

                   +---------------------+
                   |         NGFW        |
   +-------------- +-----------+---------+
   |               |           |         |
   |DUT Features   | Mandatory | Optional|
   |               |           |         |
   +-------------------------------------+


   o  Detection of CVEs matching the following characteristics when
[acm] Spell-out at first use CVE = Cert. Validation E???
      searching the National Vulnerability Database (NVD)

   different categories; namely small, medium, and maximum.
[acm] ...in four categories...


4.3.1.1.  TCP Stack Attributes

   The TCP stack SHOULD use a TCP Reno [RFC5681] variant, which include
   congestion avoidance, back off and windowing, fast retransmission,
   and fast recovery on every TCP connection between client and server
   endpoints.  The default IPv4 and IPv6 MSS segments size MUST be set
   to 1460 bytes and 1440 bytes respectively and a TX and RX receive
   windows of 65536 bytes.  Client initial congestion window MUST NOT
[acm] 
How has the TCP window discussion on BMWG-list been considered here?

   The following equation can be used to determine the required total
   number of client IP address.
[acm] s/address/addresses/

...
   Based on deployment and use case scenario, the value for "Throughput
   per IP address" can be varied.

   (Option 1)  Enterprise customer use case: 6-7 Mbps per IP (e.g.
               1,400-1,700 IPs per 10Gbit/s throughput)

   (Option 2)  Mobile ISP use case: 0.1-0.2 Mbps per IP (e.g.
               50,000-100,000 IPs per 10Gbit/s throughput)

   Based on deployment and use case scenario, client IP addresses SHOULD
   be distributed between IPv4 and IPv6 type.  The Following options can
   be considered for a selection of traffic mix ratio.
[acm] don't have two "Option 1" with different meanings. Option Mobile ???
   (Option 1)  100 % IPv4, no IPv6

   (Option 2)  80 % IPv4, 20% IPv6

...

   For encrypted traffic, the following attributes SHALL define the
   negotiated encryption parameters.  The test clients MUST use TLSv1.2
   or higher.  TLS record size MAY be optimized for the HTTPS response
   object size up to a record size of 16 KByte.  The client endpoint
   MUST send TLS Extension Server Name Indication (SNI) information when
   opening a security tunnel.  Each client connection MUST perform a
   full handshake with server certificate and MUST NOT use session reuse
[acm] missing space "servercertificate"


4.3.2.  Backend Server Configuration

   This document specifies which parameters should be considerable while
[acm] s/considerable/considered/
   configuring emulated backend servers using test equipment.

...
   2.  In the ramp up phase, the test equipment SHOULD start to generate
       the test traffic.  It SHOULD use a set approximate number of
       unique client IP addresses actively to generate traffic.  The
       traffic SHOULD ramp from zero to desired target objective.  The
       target objective will be defined for each benchmarking test.  The
[acm] predefined, and where? target not mentioned before...


   Test bed reference pre-tests help to ensure that the desired traffic
   generator aspects such as maximum throughput and the network
   performance metrics such as maximum latency and maximum packet loss
   are met.
[acm] does this mean that the traffic gen + receiver might have 
limits on the max latency or loss it can measure?
OR, that the traffic gen + receiver can measure the 
max latency and packet loss accurately? 

   Once the desired maximum performance goals for the system under test


...
       C.  DUT Enabled Features

           +  Specific features, such as logging, NGFW, DPI MUST be
              documented
[acm] specific features *used* ??


   4.  Results Summary / Executive Summary

       1.  Results SHOULD resemble a pyramid in how it is reported, with
           the introduction section documenting the summary of results
           in a prominent, easy to read block.

       2.  In the result section of the test report, the following
           attributes should be present for each test scenario.

           a.  KPIs MUST be documented separately for each test
               scenario.  The format of the KPI metrics should be
               presented as described in Section 6.1.
[acm] Are some KPIs Benchmarks?

           b.  The next level of details SHOULD be graphs showing each
               of these metrics over the duration (sustain phase) of the
               test.  This allows the user to see the measured
[acm] see *if* the measured performance is stable over time.  ??
               performance stability changes over time.

6.1.  Key Performance Indicators

   This section lists KPIs for overall benchmarking tests scenarios.
   All KPIs MUST be measured during the sustain phase of the traffic
   load profile described in Section 4.3.4.  All KPIs MUST be measured
   from the result output of test equipment.

   o  Concurrent TCP Connections
      This key performance indicator measures the average concurrent
      open TCP connections in the sustaining period.
[acm] This is a Capacity Benchmark, IMO.

...

   o  Time to First Byte (TTFB)
      This key performance indicator will measure minimum, average and
      maximum the time to first byte.  TTFB is the elapsed time between
      sending the SYN packet from the client and receiving the first
      byte of application date from the DUT/SUT.  TTFB SHOULD be
      expressed in millisecond.
[acm]
I recommend organizing these benchmarks (KPIs) using the 
3x3 Matrix described in 
https://tools.ietf.org/html/rfc8172#section-4.4
Placing the benchmarks in the Table may reveal gaps 
that need to be addressed now, or in the future.


7.  Benchmarking Tests

7.1.  Throughput Performance With NetSecOPEN Traffic Mix
[acm] What's the Trademark situation for NetSecOpen,
and all the names listed in Appendix A???

...

      One of the following ciphers and keys are RECOMMENDED to use for
      this test scenarios.

      1.  ECHDE-ECDSA-AES128-GCM-SHA256 with Prime256v1 (Signature Hash
          Algorithm: ecdsa_secp256r1_sha256 and Supported group:
          sepc256r1)

      2.  ECDHE-RSA-AES128-GCM-SHA256 with RSA 2048 (Signature Hash
          Algorithm: rsa_pkcs1_sha256 and Supported group: sepc256)

      3.  ECDHE-ECDSA-AES256-GCM-SHA384 with Secp521 (Signature Hash
          Algorithm: ecdsa_secp384r1_sha384 and Supported group:
          sepc521r1)

      4.  ECDHE-RSA-AES256-GCM-SHA384 with RSA 4096 (Signature Hash
          Algorithm: rsa_pkcs1_sha384 and Supported group: secp256)
[acm] My guess is that someone will require that we cite a reference
for all of these ciphers...


7.1.3.4.  Test Results Acceptance Criteria
[acm] IMO, Acceptance is the right action, but the wrong word
because these are not "Acceptance Tests".  I think something like
        Test Validation Criteria  will work here.

   The following test Criteria is defined as test results acceptance
[acm] s/acceptance/validation/ everywhere...
   criteria.  Test results acceptance criteria MUST be monitored during
   the whole sustain phase of the traffic load profile.

   a.  Number of failed Application transactions MUST be less than
       0.001% (1 out of 100,000 transactions) of total attempt
       transactions

...

7.2.4.3.  Step 3: Test Iteration

   Determine the maximum and average achievable connections per second
   within the acceptance criteria.
[acm] how? with additional tests using all 5 phases?


...

7.4.3.4.  Measurement

   Following KPI metrics MUST be reported for each test scenario and
   HTTP response object sizes separately:

   average TCP connections per second and average application
   transaction latency
[acm] So, this is the Average of application transaction times
meeting the validation criteria, which does not limit the time variation?
Seems like a measure of the transaction time range is needed, too.

...
7.5.3.3.  Test Results Acceptance Criteria
...
   d.  During the sustain phase, the maximum deviation (max. dev) of
       application transaction latency or TTLB (Time To Last Byte) MUST
       be less than 10%
[acm] Why doesn't criteria (d.) appear in 
7.4.TCP/HTTP Transaction Latency ??