Re: [bmwg] Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-14: (with DISCUSS and COMMENT)

[Roman Danyliw <[e-mail]>]

Hi Bala!

From: Bala Balarajah <[e-mail]>
Sent: Wednesday, October 19, 2022 6:42 PM
To: Roman Danyliw <[e-mail]>
Cc: The IESG <[e-mail]>; [e-mail]; [e-mail]; [e-mail]; Al Morton <[e-mail]>
Subject: Re: Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-14: (with DISCUSS and COMMENT)

Hi Roman,

Thanks for the review. Please see our responses inline below. If you are satisfied with our responses, we will post the new draft version before IETF 115 draft cutoff (Monday, Oct 24th ).


----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

(Updated Ballot)

-- [per -13] Recognizing that NGFW, NGIPS and UTM are not precise product
categories, offerings in this space commonly rely on statistical models or AI
techniques (e.g., machine learning) to improve detection rates and reduce false
positives to realize the capabilities in Table 1 and 2.  If even possible, how
should these settings be tuned?  How should the training period be handled when
describing the steps of the test regime (e.g., in Section 4.3.4? Section 7.2.4?)

[per -14] Thank for explaining that the training phase would not be included in
the threat emulating in your email response.  Since the goal of these document
is specify reproducible testing, the primary text I was look for was an
acknowledgment that the detection performance of some systems may be affected
by learning from prior traffic.  Any state kept by such systems much be reset
between testing runs.

[Authors] :  Machine Learning and behavioral analysis systems are not included in the scope of this test, as it uses lab-generated traffic for measurement of performance KPIs, and captured/replayed traffic as the body of the security portion of testing. Neither of these environments is conducive to the use of ML or behavioral analysis solutions.
We can add the following sentence in the draft, if it gives more clarity:
"Machine Learning and behavioral analysis features are not included in the scope of the performance benchmarking test."

Thanks.  Your proposed text and assessment seem a little bit different.  The latter is more of what I expected, these systems are out of scope).  The former seems to say something weaker, assessing these features aren’t in scope.  My recommendation would be for something a bit stronger on what to do with a this testing regime relative to this class of system appended to the end of Section 3 that combines both the former and latter ideas.  Roughly:

==[ snip ]==

The performance testing methodology described in this document is not intended for devices that rely on machine learning or behavioral analysis.  If such features are present in a device under test, they should be disabled.

==[ snip ]==



----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

(Updated Ballot)

Thanks for the changes made in -13.

** [per -13] Section 3. Per “This document focuses on advanced, …”, what makes
a testing method “advanced”?

[Authors]: Comparing previous RFCs 2544 and RFC3511, this draft provides a more in-depth test methodology for test parameter definition, test results validation criteria, and test procedures defined in section 7 and its subsections.

[Roman] Makes sense.  My intent was to suggest that this distinction from RFC2544/3511 being described as advanced was not clear.


** [per -13] Section 4.2.  Should the following additional features be noted as
a feature of NGFWs and NGIPS (Table 2 and 3 in -14)?

-- geolocation or network topology-based classification/filtering (since there
is normative text “Geographical location filtering SHOULD be configured.”)

[Authors]:   We will add the following sentence in the next release:
Geographical location filtering SHOULD be configured. If the DUT/SUT is not designed to perform geographical location filtering, it is acceptable to conduct tests without them.  However, this MUST be
noted in the test report.

[Roman] Thanks.


** [per -13/14] Table 2.  Is there a Anti-Evasion (listed in Table 3 for NGIPS)
are not mentioned here (for NGFW).

[Authors]:  Anti-Evasion should be included in NGFW in the same manner as NGIPS. We will add this in the next release.

[Roman] Thanks.


** [per -13] Section 4.2.  Per “Logging SHOULD be enabled.”  How does this
“SHOULD” align with “logging and reporting” being a RECOMMENDED in Table 1 and
2?

[Authors]: According to the security product vendors (the draft contributors), "logging and reporting" is one of the mandatory (MUST) and default features for security devices. For this reason, we removed it from the tables that contain RECOMMENDED and
OPTIONAL features only. Therefore, we added the following text below table 3, which applies to both NGFW and NGIPS:
“Logging and reporting MUST be enabled."

[Roman] The way I was approach it was with the expectation that all of the possible features described in Table 1 were going to be crossed walked with their normative requirements in Table 2 and 3.  If the WG prefers to keep it as described, no problem.  Thanks for explaining.


[per -14]  Thanks for the edits here.  I think a regression was a regression
introduced.  Table 3 (NGIPS) used to have “Logging and Reporting” just like
Table 2 in -12.

[Authors]:  There was a mistake. As mentioned above, "Logging and Reporting" will be removed from both tables. We will update this in the next release.

[Roman] Thanks.  It was the inconsistency that caught me.

Thanks,
Roman