Re: [bmwg] Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-14: (with DISCUSS and COMMENT)

[Bala Balarajah <[e-mail]>]

Hi Roman,

Thank you for your recommended text. We have updated Section 3 and also
incorporated all your comments into the new version. I have just posted the
new version 15.

Thanks,
Bala

Am Do., 20. Okt. 2022 um 19:13 Uhr schrieb Roman Danyliw <[e-mail]>:

> Hi Bala!
>
>
>
> *From:* Bala Balarajah <[e-mail]>
> *Sent:* Wednesday, October 19, 2022 6:42 PM
> *To:* Roman Danyliw <[e-mail]>
> *Cc:* The IESG <[e-mail]>; [e-mail];
> [e-mail]; [e-mail]; Al Morton <[e-mail]>
> *Subject:* Re: Roman Danyliw's Discuss on
> draft-ietf-bmwg-ngfw-performance-14: (with DISCUSS and COMMENT)
>
>
>
> Hi Roman,
>
>
>
> Thanks for the review. Please see our responses inline below. If you are
> satisfied with our responses, we will post the new draft version before
> IETF 115 draft cutoff (Monday, Oct 24th ).
>
>
>
>
>
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
>
> (Updated Ballot)
>
> -- [per -13] Recognizing that NGFW, NGIPS and UTM are not precise product
> categories, offerings in this space commonly rely on statistical models or
> AI
> techniques (e.g., machine learning) to improve detection rates and reduce
> false
> positives to realize the capabilities in Table 1 and 2.  If even possible,
> how
> should these settings be tuned?  How should the training period be handled
> when
> describing the steps of the test regime (e.g., in Section 4.3.4? Section
> 7.2.4?)
>
> [per -14] Thank for explaining that the training phase would not be
> included in
> the threat emulating in your email response.  Since the goal of these
> document
> is specify reproducible testing, the primary text I was look for was an
> acknowledgment that the detection performance of some systems may be
> affected
> by learning from prior traffic.  Any state kept by such systems much be
> reset
> between testing runs.
>
>
>
> [Authors] :  Machine Learning and behavioral analysis systems are not
> included in the scope of this test, as it uses lab-generated traffic for
> measurement of performance KPIs, and captured/replayed traffic as the body
> of the security portion of testing. Neither of these environments is
> conducive to the use of ML or behavioral analysis solutions.
>
> We can add the following sentence in the draft, if it gives more clarity:
>
> "Machine Learning and behavioral analysis features are not included in the
> scope of the performance benchmarking test."
>
>
>
> Thanks.  Your proposed text and assessment seem a little bit different.
> The latter is more of what I expected, these systems are out of scope).
> The former seems to say something weaker, assessing these features aren’t
> in scope.  My recommendation would be for something a bit stronger on what
> to do with a this testing regime relative to this class of system appended
> to the end of Section 3 that combines both the former and latter ideas.
> Roughly:
>
>
>
> ==[ snip ]==
>
>
>
> The performance testing methodology described in this document is not
> intended for devices that rely on machine learning or behavioral analysis.
> If such features are present in a device under test, they should be
> disabled.
>
>
>
> ==[ snip ]==
>
>
>
>
>
>
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
>
> (Updated Ballot)
>
> Thanks for the changes made in -13.
>
> ** [per -13] Section 3. Per “This document focuses on advanced, …”, what
> makes
> a testing method “advanced”?
>
>
>
> [Authors]: Comparing previous RFCs 2544 and RFC3511, this draft provides a
> more in-depth test methodology for test parameter definition, test results
> validation criteria, and test procedures defined in section 7 and its
> subsections.
>
>
>
> [Roman] Makes sense.  My intent was to suggest that this distinction from
> RFC2544/3511 being described as advanced was not clear.
>
>
>
>
> ** [per -13] Section 4.2.  Should the following additional features be
> noted as
> a feature of NGFWs and NGIPS (Table 2 and 3 in -14)?
>
> -- geolocation or network topology-based classification/filtering (since
> there
> is normative text “Geographical location filtering SHOULD be configured.”)
>
>
>
> [Authors]:   We will add the following sentence in the next release:
>
> Geographical location filtering SHOULD be configured. If the DUT/SUT is
> not designed to perform geographical location filtering, it is acceptable
> to conduct tests without them.  However, this MUST be
> noted in the test report.
>
>
>
> [Roman] Thanks.
>
>
>
>
> ** [per -13/14] Table 2.  Is there a Anti-Evasion (listed in Table 3 for
> NGIPS)
> are not mentioned here (for NGFW).
>
>
>
> [Authors]:  Anti-Evasion should be included in NGFW in the same manner as
> NGIPS. We will add this in the next release.
>
>
>
> [Roman] Thanks.
>
>
>
>
> ** [per -13] Section 4.2.  Per “Logging SHOULD be enabled.”  How does this
> “SHOULD” align with “logging and reporting” being a RECOMMENDED in Table 1
> and
> 2?
>
>
>
> [Authors]: According to the security product vendors (the draft
> contributors), "logging and reporting" is one of the mandatory (MUST) and
> default features for security devices. For this reason, we removed it from
> the tables that contain RECOMMENDED and
>
> OPTIONAL features only. Therefore, we added the following text below table
> 3, which applies to both NGFW and NGIPS:
>
> “Logging and reporting MUST be enabled."
>
>
>
> [Roman] The way I was approach it was with the expectation that all of the
> possible features described in Table 1 were going to be crossed walked with
> their normative requirements in Table 2 and 3.  If the WG prefers to keep
> it as described, no problem.  Thanks for explaining.
>
>
>
>
> [per -14]  Thanks for the edits here.  I think a regression was a
> regression
> introduced.  Table 3 (NGIPS) used to have “Logging and Reporting” just like
> Table 2 in -12.
>
>
>
> [Authors]:  There was a mistake. As mentioned above, "Logging and
> Reporting" will be removed from both tables. We will update this in the
> next release.
>
>
>
> [Roman] Thanks.  It was the inconsistency that caught me.
>
>
>
> Thanks,
>
> Roman
>