Re: [bmwg] Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

[[e-mail]]

Roman,

Do you have response to the questions Carsten asked in response to you?

Brian

-----Original Message-----
From: Carsten Rossenhoevel <[e-mail]> 
Sent: Wednesday, February 2, 2022 7:48 PM
To: Roman Danyliw <[e-mail]>; The IESG <[e-mail]>
Cc: [e-mail]; [e-mail]; [e-mail]; Al Morton <[e-mail]>
Subject: Re: Roman Danyliw's Discuss on draft-ietf-bmwg-ngfw-performance-13: (with DISCUSS and COMMENT)

Dear Roman,

thank you for your comments and questions. Primarily, the document is focused on performance benchmarking as mentioned in 4.2.1.  This is the area where the authors and contributors saw a major gap in test methodology.  Today's datasheets of commercial firewalls are sometimes difficult to interpret.  Performance benchmarking methodology applies across all types of security features (per Table 1).

Thus, we describe performance benchmarking methodology in great detail. Sections 4.3 and 7 focus on representative and reproducible load profiles.  (Note - Training phases would not include threat emulation. This is normal and not mentioned otherwise in the document - do you recommend we change any text / add an explanation?)

Advanced security features use a wide range of functions and methods, as you mentioned.  It would be very difficult (and a race against time) to cover white-box test methodology exhaustively for such security features in a standards document.  Therefore in this document, we:

a) describe the recommended and optional security features at high level (section 4.2)

b) suggest evaluating the correct functional security effectiveness of these features in advance of performance benchmarking, outside of the core performance benchmarking methodology (mentioned in section 4.2.1, second paragraph)

c) safeguard against the false testing of a firewall without proper security features enabled (mentioned in section 4.2.1, first paragraph and 2nd sentence in second paragraph)

d) describe security effectiveness test methodology verifying protection against common vulnerabilities and exposures (CVEs) in Appendix A.  This test methodology is a black-box approach not aiming to verify/compare advanced concepts such as sandboxing, zero-day threat protection, and other AI-based functions.   We are working on test methodology for these areas in the NetSecOPEN group; but it is difficult to come up with reasonable test solutions (specifically if they should be reproducible and test tool-agnostic), which is why this document does not include detailed methodology yet.


Regarding "CVE" nomenclature, I agree that the language in Appendix A is not absolutely precise.  The term CVE has become informally used for the
(NIST/Mitre) database entries, the firewall capability to protect, and for the test tool attack type.   The authors are open to improving the terminology.  What do you suggest (without making each sentence a mouthful)?

I also agree with your observation that CVEs are often not atomic and that test tool implementations and firewall implementations may cover parts of a CVE.  This is an issue of the CVE database - there is no consistent numbering of sub-CVEs.  We have suffered from this issue in our security effectiveness methodology verification program we have conducted since early 2021.  Our goal is to create a standard document that is not tied to any specific test tool or lab implementation.  Two test labs (UNH-IOL and EANTC) have worked with two test tool vendors (Spirent and Keysight) and multiple firewall vendors.  We  have evaluated thousands of CVEs, comparing test coverage and results.

Each test tool vendor uses individual implementations to emulate threats aligned with CVEs, and their implementations sometimes differ in the sub-CVE coverage.  Firewall vendors also have their individual code and do not always cover all sub-CVEs - often for good reasons that need to be analyzed on the application layer individually (very high effort). We concluded that we have to accept the state of the art in CVE granularity and that we have to live with the uncertainties of sub-CVE coverage in general.  (In detail we will continue to work with test tool and firewall vendors to align implementations and aim for best-working CVE threat protection.)  If you have any ideas how this draft document could help improve the industry more quickly, we are seriously open for suggestions.


Re your other comments:

- The introduction section and tables 1/2/3 are being rephrased editorially to increase their readability and add more explanations, based on feedback from other AD reviewers.

- In section 4.2, I am not sure if I understand your comment about logging.  According to RFC 2119, "RECOMMEND" and "SHOULD" consistently describe the same level of requirement?

- Well-defined TCP stack attributes are required for a network security device because transport layer attributes influence its performance greatly.  Many issues found in EANTC tests in the past originated from incorrect TCP stack attributes, and non-reproducible behavior of a firewall is often related to undocumented/uncontrolled TCP stack attributes.

- From our understanding, TLS 1.3 cipher suites are all considered adequate in the industry today, while some of the TLS 1.2 cipher suites are considered deprecated. This is why we specify them in detail.  That said, the note below the list of TLS 1.2 recommended ciphers encourages the use of ciphers appropriate for evolving use cases.  This might lead to a selection of specific TLS 1.3 ciphers in the future as well.

- Good point regarding the use of emulated exploits in section 9.  We agree to add a note.

- Regarding 2nd paragraph of A.1, second sentence ("In parallel..."), we agree that the text is imprecise.  Threats that apply to the connection setup cannot be encrypted.  We'll edit the text to apply to these cases as well.

Best regards, Carsten



Am 2/2/2022 um 11:39 PM schrieb Roman Danyliw via Datatracker:
> Roman Danyliw has entered the following ballot position for
> draft-ietf-bmwg-ngfw-performance-13: Discuss
>
> When responding, please keep the subject line intact and reply to all 
> email addresses included in the To and CC lines. (Feel free to cut 
> this introductory paragraph, however.)
>
>
> Please refer to 
> https://www.ietf.org/blog/handling-iesg-ballot-positions/
> for more information about how to handle DISCUSS and COMMENT positions.
>
>
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-bmwg-ngfw-performance/
>
>
>
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
>
> ** A key element of successfully running the throughput tests 
> described in Section 7, appears to be ensuring how to configure the device under test.
> Section 4.2. helpfully specifies feature sets with recommendations 
> configurations.  However, it appears there are elements of 
> under-specification given the level of detail specified with normative language.  Specifically:
>
> -- Section 4.2.1 seems unspecified regarding all the capabilities in 
> Table 1 and 2.  The discussion around vulnerabilities (CVEs) does not 
> appear to be relevant to configuration of anti-spyware, anti-virus, 
> anti-botnet, DLP, and DDOS.
>
> -- Recognizing that NGFW, NGIPS and UTM are not precise product 
> categories, offerings in this space commonly rely on statistical 
> models or AI techniques (e.g., machine learning) to improve detection 
> rates and reduce false positives to realize the capabilities in Table 
> 1 and 2.  If even possible, how should these settings be tuned?  How 
> should the training period be handled when describing the steps of the 
> test regime (e.g., in Section 4.3.4? Section 7.2.4?)
>
> ** Appendix A.  The KPI measures don’t seem precise here – CVEs are 
> unlikely to be the measure seen on the wire.  Wouldn’t it be exploits 
> associated with a particular vulnerability (that’s numbered via CVE)?  
> There can be a one-to-many relationship between the vulnerability and 
> exploits (e.g., multiple products affected by a single CVE); or the multiple implementations of an exploit.
>
>
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
>
> ** Abstract.  NGFW, NGIPS and UTM are fuzzy product categories.  Do 
> you want to them somewhere?  How do they differ in functionality?  UTM 
> is mentioned here, but not again in the document.
>
> ** Section 1.
> The requirements
>     for network security element performance and effectiveness have
>     increased tremendously since then.  In the eighteen years since
>     [RFC3511] was published, recommending test methodology and
>     terminology for firewalls, requirements and expectations for network
>     security elements has increased tremendously.
>
> I don’t follow how the intent of these two sentences is different.  
> Given the other text in this paragraph, these sentences also appear redundant.
>
> ** Section 3. Per “This document focuses on advanced, …”, what makes a 
> testing method “advanced”?
>
> ** Section 4.2.  The abstract said that testing for NGFW, NGIPS and 
> UTM would be provided.  This section is silent on UTM.
>
> ** Section 4.2.  Should the following additional features be noted as 
> a feature of NGFWs and NGIPS (Tables 1 and 2)?
>
> -- reconnaissance detection
>
> -- geolocation or network topology-based classification/filtering
>
> ** Section 4.2. Thanks for the capability taxonomies describe here.  
> Should it be noted that “Table 1 and 2 are approximate taxonomies of 
> features commonly found in currently deployed NGFW and NGIDS.  The 
> features provided by specific implementations may be named differently 
> and not necessarily have configuration settings that align to the taxonomy.”
>
> ** Table 1.  Is there a reason that DPI and Anti-Evasion (listed in 
> Table 2 for
> NGIPS) are not mentioned here (for NGFW).  I don’t see how many (all?) 
> of the features listed as RECOMMENDED could be done without it.
>
> ** Table 3.  For Anti-Botnet, should it read “detects and blocks”?
>
> ** Table 3.  For Web Filtering, is this scoped to be classification 
> and threat detection by URI?
>
> ** Table 3.  This table is missing a description for DoS from Table 1 
> and DPI and Anti-Evasion from Table 2.
>
> ** Section 4.2.  Per “Logging SHOULD be enabled.”  How does this 
> “SHOULD” align with “logging and reporting” being a RECOMMENDED in 
> Table 1 and 2?  Same question on “Application Identification and Control SHOULD be configured”
>
> ** Section 4.3.1.1.  Why is such well-formed and well-behaved traffic 
> assumed for a security device?
>
> ** Section 4.3.1.  What cipher suites should be used for TLS 1.3 based tests?
> The text is prescriptive for TLS 1.2 (using a RECOMMEND) but simply 
> restates all of those registered by RFC8446.
>
> ** Section 9.  Given that the configurations of these test will 
> include working exploits, it would be helpful to provide a reminder on 
> the need control access to them.
>
> ** Section A.1.
> In parallel, the CVEs will be sent to the DUT/SUT as
>     encrypted and as well as clear text payload formats using a traffic
>     generator.
>
> This guidance doesn’t seem appropriate for all cases.  Couldn’t the 
> vulnerability being exploited involve a payload in the unencrypted 
> part or a phase in the communication exchange before a secure channel is negotiated?
>
> ** Editorial nits
> -- Section 1.  Editorial. s/for firewalls initially/for firewalls/
>
> -- Section 5.  Typo. s/as test equipments/as test equipment/
>
>
>
--
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center) Salzufer 14, 10587 Berlin, Germany office +49.30.3180595-21, fax [phone], mobile [phone] [e-mail], https://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany EU VAT No: DE812824025