Re: [bmwg] Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-14: (with DISCUSS and COMMENT)

[Bala Balarajah <[e-mail]>]

Hi Roman,

Thanks for the review. Please see our responses inline below. If you are
satisfied with our responses, we will post the new draft version before
IETF 115 draft cutoff (Monday, Oct 24th ).



> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
>
> (Updated Ballot)
>
> -- [per -13] Recognizing that NGFW, NGIPS and UTM are not precise product
> categories, offerings in this space commonly rely on statistical models or
> AI
> techniques (e.g., machine learning) to improve detection rates and reduce
> false
> positives to realize the capabilities in Table 1 and 2.  If even possible,
> how
> should these settings be tuned?  How should the training period be handled
> when
> describing the steps of the test regime (e.g., in Section 4.3.4? Section
> 7.2.4?)
>
> [per -14] Thank for explaining that the training phase would not be
> included in
> the threat emulating in your email response.  Since the goal of these
> document
> is specify reproducible testing, the primary text I was look for was an
> acknowledgment that the detection performance of some systems may be
> affected
> by learning from prior traffic.  Any state kept by such systems much be
> reset
> between testing runs.
>
>
[Authors] :  Machine Learning and behavioral analysis systems are not
included in the scope of this test, as it uses lab-generated traffic for
measurement of performance KPIs, and captured/replayed traffic as the body
of the security portion of testing. Neither of these environments is
conducive to the use of ML or behavioral analysis solutions.
We can add the following sentence in the draft, if it gives more clarity:
"Machine Learning and behavioral analysis features are not included in the
scope of the performance benchmarking test."


> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
>
> (Updated Ballot)
>
> Thanks for the changes made in -13.
>
> ** [per -13] Section 3. Per “This document focuses on advanced, …”, what
> makes
> a testing method “advanced”?
>

[Authors]: Comparing previous RFCs 2544 and RFC3511, this draft provides a
more in-depth test methodology for test parameter definition, test results
validation criteria, and test procedures defined in section 7 and its
subsections.


>
> ** [per -13] Section 4.2.  Should the following additional features be
> noted as
> a feature of NGFWs and NGIPS (Table 2 and 3 in -14)?
>
> -- geolocation or network topology-based classification/filtering (since
> there
> is normative text “Geographical location filtering SHOULD be configured.”)
>

[Authors]:   We will add the following sentence in the next release:
Geographical location filtering SHOULD be configured. If the DUT/SUT is not
designed to perform geographical location filtering, it is acceptable to
conduct tests without them.  However, this MUST be
noted in the test report.


> ** [per -13/14] Table 2.  Is there a Anti-Evasion (listed in Table 3 for
> NGIPS)
> are not mentioned here (for NGFW).
>

[Authors]:  Anti-Evasion should be included in NGFW in the same manner as
NGIPS. We will add this in the next release.


> ** [per -13] Section 4.2.  Per “Logging SHOULD be enabled.”  How does this
> “SHOULD” align with “logging and reporting” being a RECOMMENDED in Table 1
> and
> 2?
>

[Authors]: According to the security product vendors (the draft
contributors), "logging and reporting" is one of the mandatory (MUST) and
default features for security devices. For this reason, we removed it from
the tables that contain RECOMMENDED and OPTIONAL features only. Therefore,
we added the following text below table 3, which applies to both NGFW and
NGIPS:
“Logging and reporting MUST be enabled."


> [per -14]  Thanks for the edits here.  I think a regression was a
> regression
> introduced.  Table 3 (NGIPS) used to have “Logging and Reporting” just like
> Table 2 in -12.
>

[Authors]:  There was a mistake. As mentioned above, "Logging and
Reporting" will be removed from both tables. We will update this in the
next release.


Best Regards,
Bala