Re: [bmwg] Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

[Carsten Rossenhoevel <[e-mail]>]

Dear Roman,

thank you for your comments and questions. Primarily, the document is 
focused on performance benchmarking as mentioned in 4.2.1.  This is the 
area where the authors and contributors saw a major gap in test 
methodology.  Today's datasheets of commercial firewalls are sometimes 
difficult to interpret.  Performance benchmarking methodology applies 
across all types of security features (per Table 1).

Thus, we describe performance benchmarking methodology in great detail.  
Sections 4.3 and 7 focus on representative and reproducible load 
profiles.  (Note - Training phases would not include threat emulation.  
This is normal and not mentioned otherwise in the document - do you 
recommend we change any text / add an explanation?)

Advanced security features use a wide range of functions and methods, as 
you mentioned.  It would be very difficult (and a race against time) to 
cover white-box test methodology exhaustively for such security features 
in a standards document.  Therefore in this document, we:

a) describe the recommended and optional security features at high level 
(section 4.2)

b) suggest evaluating the correct functional security effectiveness of 
these features in advance of performance benchmarking, outside of the 
core performance benchmarking methodology (mentioned in section 4.2.1, 
second paragraph)

c) safeguard against the false testing of a firewall without proper 
security features enabled (mentioned in section 4.2.1, first paragraph 
and 2nd sentence in second paragraph)

d) describe security effectiveness test methodology verifying protection 
against common vulnerabilities and exposures (CVEs) in Appendix A.  This 
test methodology is a black-box approach not aiming to verify/compare 
advanced concepts such as sandboxing, zero-day threat protection, and 
other AI-based functions.   We are working on test methodology for these 
areas in the NetSecOPEN group; but it is difficult to come up with 
reasonable test solutions (specifically if they should be reproducible 
and test tool-agnostic), which is why this document does not include 
detailed methodology yet.


Regarding "CVE" nomenclature, I agree that the language in Appendix A is 
not absolutely precise.  The term CVE has become informally used for the 
(NIST/Mitre) database entries, the firewall capability to protect, and 
for the test tool attack type.   The authors are open to improving the 
terminology.  What do you suggest (without making each sentence a mouthful)?

I also agree with your observation that CVEs are often not atomic and 
that test tool implementations and firewall implementations may cover 
parts of a CVE.  This is an issue of the CVE database - there is no 
consistent numbering of sub-CVEs.  We have suffered from this issue in 
our security effectiveness methodology verification program we have 
conducted since early 2021.  Our goal is to create a standard document 
that is not tied to any specific test tool or lab implementation.  Two 
test labs (UNH-IOL and EANTC) have worked with two test tool vendors 
(Spirent and Keysight) and multiple firewall vendors.  We  have 
evaluated thousands of CVEs, comparing test coverage and results.

Each test tool vendor uses individual implementations to emulate threats 
aligned with CVEs, and their implementations sometimes differ in the 
sub-CVE coverage.  Firewall vendors also have their individual code and 
do not always cover all sub-CVEs - often for good reasons that need to 
be analyzed on the application layer individually (very high effort).  
We concluded that we have to accept the state of the art in CVE 
granularity and that we have to live with the uncertainties of sub-CVE 
coverage in general.  (In detail we will continue to work with test tool 
and firewall vendors to align implementations and aim for best-working 
CVE threat protection.)  If you have any ideas how this draft document 
could help improve the industry more quickly, we are seriously open for 
suggestions.


Re your other comments:

- The introduction section and tables 1/2/3 are being rephrased 
editorially to increase their readability and add more explanations, 
based on feedback from other AD reviewers.

- In section 4.2, I am not sure if I understand your comment about 
logging.  According to RFC 2119, "RECOMMEND" and "SHOULD" consistently 
describe the same level of requirement?

- Well-defined TCP stack attributes are required for a network security 
device because transport layer attributes influence its performance 
greatly.  Many issues found in EANTC tests in the past originated from 
incorrect TCP stack attributes, and non-reproducible behavior of a 
firewall is often related to undocumented/uncontrolled TCP stack attributes.

- From our understanding, TLS 1.3 cipher suites are all considered 
adequate in the industry today, while some of the TLS 1.2 cipher suites 
are considered deprecated. This is why we specify them in detail.  That 
said, the note below the list of TLS 1.2 recommended ciphers encourages 
the use of ciphers appropriate for evolving use cases.  This might lead 
to a selection of specific TLS 1.3 ciphers in the future as well.

- Good point regarding the use of emulated exploits in section 9.  We 
agree to add a note.

- Regarding 2nd paragraph of A.1, second sentence ("In parallel..."), we 
agree that the text is imprecise.  Threats that apply to the connection 
setup cannot be encrypted.  We'll edit the text to apply to these cases 
as well.

Best regards, Carsten



Am 2/2/2022 um 11:39 PM schrieb Roman Danyliw via Datatracker:
> Roman Danyliw has entered the following ballot position for
> draft-ietf-bmwg-ngfw-performance-13: Discuss
>
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
>
>
> Please refer to https://www.ietf.org/blog/handling-iesg-ballot-positions/
> for more information about how to handle DISCUSS and COMMENT positions.
>
>
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-bmwg-ngfw-performance/
>
>
>
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
>
> ** A key element of successfully running the throughput tests described in
> Section 7, appears to be ensuring how to configure the device under test.
> Section 4.2. helpfully specifies feature sets with recommendations
> configurations.  However, it appears there are elements of under-specification
> given the level of detail specified with normative language.  Specifically:
>
> -- Section 4.2.1 seems unspecified regarding all the capabilities in Table 1
> and 2.  The discussion around vulnerabilities (CVEs) does not appear to be
> relevant to configuration of anti-spyware, anti-virus, anti-botnet, DLP, and
> DDOS.
>
> -- Recognizing that NGFW, NGIPS and UTM are not precise product categories,
> offerings in this space commonly rely on statistical models or AI techniques
> (e.g., machine learning) to improve detection rates and reduce false positives
> to realize the capabilities in Table 1 and 2.  If even possible, how should
> these settings be tuned?  How should the training period be handled when
> describing the steps of the test regime (e.g., in Section 4.3.4? Section 7.2.4?)
>
> ** Appendix A.  The KPI measures don’t seem precise here – CVEs are unlikely to
> be the measure seen on the wire.  Wouldn’t it be exploits associated with a
> particular vulnerability (that’s numbered via CVE)?  There can be a one-to-many
> relationship between the vulnerability and exploits (e.g., multiple products
> affected by a single CVE); or the multiple implementations of an exploit.
>
>
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
>
> ** Abstract.  NGFW, NGIPS and UTM are fuzzy product categories.  Do you want to
> them somewhere?  How do they differ in functionality?  UTM is mentioned here,
> but not again in the document.
>
> ** Section 1.
> The requirements
>     for network security element performance and effectiveness have
>     increased tremendously since then.  In the eighteen years since
>     [RFC3511] was published, recommending test methodology and
>     terminology for firewalls, requirements and expectations for network
>     security elements has increased tremendously.
>
> I don’t follow how the intent of these two sentences is different.  Given the
> other text in this paragraph, these sentences also appear redundant.
>
> ** Section 3. Per “This document focuses on advanced, …”, what makes a testing
> method “advanced”?
>
> ** Section 4.2.  The abstract said that testing for NGFW, NGIPS and UTM would
> be provided.  This section is silent on UTM.
>
> ** Section 4.2.  Should the following additional features be noted as a feature
> of NGFWs and NGIPS (Tables 1 and 2)?
>
> -- reconnaissance detection
>
> -- geolocation or network topology-based classification/filtering
>
> ** Section 4.2. Thanks for the capability taxonomies describe here.  Should it
> be noted that “Table 1 and 2 are approximate taxonomies of features commonly
> found in currently deployed NGFW and NGIDS.  The features provided by specific
> implementations may be named differently and not necessarily have configuration
> settings that align to the taxonomy.”
>
> ** Table 1.  Is there a reason that DPI and Anti-Evasion (listed in Table 2 for
> NGIPS) are not mentioned here (for NGFW).  I don’t see how many (all?) of the
> features listed as RECOMMENDED could be done without it.
>
> ** Table 3.  For Anti-Botnet, should it read “detects and blocks”?
>
> ** Table 3.  For Web Filtering, is this scoped to be classification and threat
> detection by URI?
>
> ** Table 3.  This table is missing a description for DoS from Table 1 and DPI
> and Anti-Evasion from Table 2.
>
> ** Section 4.2.  Per “Logging SHOULD be enabled.”  How does this “SHOULD” align
> with “logging and reporting” being a RECOMMENDED in Table 1 and 2?  Same
> question on “Application Identification and Control SHOULD be configured”
>
> ** Section 4.3.1.1.  Why is such well-formed and well-behaved traffic assumed
> for a security device?
>
> ** Section 4.3.1.  What cipher suites should be used for TLS 1.3 based tests?
> The text is prescriptive for TLS 1.2 (using a RECOMMEND) but simply restates
> all of those registered by RFC8446.
>
> ** Section 9.  Given that the configurations of these test will include working
> exploits, it would be helpful to provide a reminder on the need control access
> to them.
>
> ** Section A.1.
> In parallel, the CVEs will be sent to the DUT/SUT as
>     encrypted and as well as clear text payload formats using a traffic
>     generator.
>
> This guidance doesn’t seem appropriate for all cases.  Couldn’t the
> vulnerability being exploited involve a payload in the unencrypted part or a
> phase in the communication exchange before a secure channel is negotiated?
>
> ** Editorial nits
> -- Section 1.  Editorial. s/for firewalls initially/for firewalls/
>
> -- Section 5.  Typo. s/as test equipments/as test equipment/
>
>
>
-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax [phone], mobile [phone]
[e-mail], https://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany
Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
EU VAT No: DE812824025