Re: [bmwg] AD Review of draft-ietf-bmwg-ngfw-performance

[Carsten Rossenhoevel <[e-mail]>]

Dear Tim,

Happy New Year!  Apologies for not responding earlier due to the 
Christmas break.

Thank you for taking the time to provide your comments/suggestions 
regarding the draft on Dec 16.

The scope of the draft certainly includes virtualized next-generation 
firewalls (NGFWs), as it is outlined in sections 5 (testbed 
considerations).  Virtualization-specific reporting requirements to 
ensure reproducibility of results are specified in section 6.

The test methodology is robust and applicable to any size and type of 
firewalls.  Large perimeter firewalls can be tested, as well as smaller 
firewalls, whether physical or virtual, whether in a private or public 
cloud.  Multi-tenancy does not require any changes to the methodology; 
in fact, it is just a choice of test traffic patterns (see section 
4.3.1.2).  To emulate traffic in multiple tenant domains, it may be 
necessary to include a router as described in Figure 2.

Sandboxing techniques are included in the "Anti-evasion" category in 
Table 1.  It is possible and supported to enable such sandboxing 
features during tests.  The test methodology remains unchanged, although 
probably the resulting performance will be different once additional 
functions are added.   As mentioned in section 4.2.1, the draft is 
primarily focused on performance benchmarking. However, it is 
recommended to validate the security features configuration of a 
solution under test by evaluating the security effectiveness as a 
prerequisite for performance benchmarking tests.  Security effectiveness 
methodology is specified in Appendix A.

Best regards, Carsten



Am 12/16/2021 um 9:59 PM schrieb tim copley:
> I also, wanted to chime in regarding this and again, I also apologize 
> for not reviewing earlier.
>
> I've not really been too involved in security devices until the last 
> couple years, but boy has it changed
> drastically from when I was.  Also, I noted that this document is 
> basically following prior standards that
> are also a bit out dated.
>
> Some of the evolution, I've noticed which I think should at least be 
> addressed in.  I think you tried
> to exclude these in the Scope.?  However I'm not really sure you are 
> covering but maybe 1/2 of the devices
> that are being deployed at this time without addressing:
>
> Virtualization. /  Remote FWs in the next gen.
>
> *) Large FWs are compartmentalizing Customers and serving multiple 
> customers with the same firewall.  So while
> you have said what would happen to a customer when The Firewall is UT, 
> you don't really address what's going
> on with all the other customers on this DUT.  If there are 10 
> Customers going through 1 large FW, what are the
> other 9 experiencing when this is under Test.  Would be interesting to 
> have multiple parallel tests underway
> and verify that behaviour is consistent across every "VDOM"  within 
> the system?
>
> *) Time to Change.  If things are changed during your Load 
> profile.perhaps even on a different virtual Firewall
> within the system,
>
> *) Sandbox dips.  Several of the firewall vendors send stuff through 
> labs / temp work spaces either online or
> offline that allow them to test against Zero Day scenarios.  What 
> happens to traffic flows during anomalies
> in the traffic flows.  Does it change the throughput of the DUT?  I 
> know this is kind of an open ended question.
> but not sure you have good benchmark stats for a FW without it...
>
> *) Cloud based scenarios.  If the firewall is removed, and I'm not 
> really sure how you would test this with
> consistent results, but if the FW is offsite and the traffic is being 
> routed through a Cloud based FW  What does
> that do to your traffic results?
>
> TimC
> Fruth Group
> [e-mail]
>
>
> On Wed, Dec 15, 2021 at 8:38 AM Warren Kumari <[e-mail]> wrote:
>
>     Hi there all,
>
>     I'd like to apologize to the authors and WG for how long it has
>     taken me to post this review; I read the document a while back,
>     but forgot to post it :-(
>
>     I only had a few editorial / readability suggestions -- they are
>     sufficiently minor (and I'm embarrassed by how long my review
>     took!) that I'll kick off LC without asking for a new version --
>     authors, please consider them if you post a new version addressing
>     LC comments, or to address IESG Eval comments.
>
>     Chairs / Al: The document state is listed as: "Doc Shepherd
>     Follow-up Underway" - I'm assuming that this is from:
>     https://mailarchive.ietf.org/arch/msg/bmwg/EonLD10nf0xQfWzHt0YT0QDK6lU
>     - can you please confirm (LOUDLY) that this is OK now and I'll
>     kick off LC.
>
>
>     W
>
>
>     1.  Introduction
>
>         18 years have passed since IETF recommended test methodology and
>         terminology for firewalls initially ([RFC3511]).  The requirements
>         for network security element performance and effectiveness have
>         increased tremendously since then.  Security function implementations
>
>     [O] 18 years have passed since IETF recommended test methodology and
>
>         terminology for firewalls initially ([RFC3511]).  The requirements
>         for network security element performance and effectiveness have
>         increased tremendously since then.
>     [P] In the eighteen years since [RFC3511] was published, recommending test methodology and terminology for firewalls, requirements and expectations for network security elements has increased tremendously.
>
>         have evolved to more advanced areas and have diversified into
>         intrusion detection and prevention, threat management, analysis of
>         encrypted traffic, etc.  In an industry of growing importance, well-
>         defined, and reproducible key performance indicators (KPIs) are
>         increasingly needed as they enable fair and reasonable comparison of
>
>     [O] as they enable
>     [P] to enable
>
>         network security functions.
>
>
>
>
>     -- 
>     Perhaps they really do strive for incomprehensibility in their specs.
>     After all, when the liturgy was in Latin, the laity knew their place.
>     -- Michael Padlipsky
>     _______________________________________________
>     bmwg mailing list
>     [e-mail]
>     https://www.ietf.org/mailman/listinfo/bmwg
>
>
>
> -- 
> Timothy Copley
> 602.350.0633

-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax [phone], mobile [phone]
[e-mail],https://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany
Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
EU VAT No: DE812824025