[bmwg] Second draft for next-gen firewall (NGFW) performance benchmarking uploaded

Carsten Rossenhoevel <cross@eantc.de> Wed, 28 February 2018 11:38 UTC

Return-Path: <cross@eantc.de>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 73005127873 for <bmwg@ietfa.amsl.com>; Wed, 28 Feb 2018 03:38:54 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Level:
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NM6-08uL-NBy for <bmwg@ietfa.amsl.com>; Wed, 28 Feb 2018 03:38:49 -0800 (PST)
Received: from obelix.eantc.de (ns.eantc.com [89.27.172.100]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8718D12785F for <bmwg@ietf.org>; Wed, 28 Feb 2018 03:38:49 -0800 (PST)
Received: from obelix.eantc.de ([192.168.100.100] helo=localhost) by obelix.eantc.de with esmtp (Exim 4.80) (envelope-from <cross@eantc.de>) id 1er04N-0006HD-3M for bmwg@ietf.org; Wed, 28 Feb 2018 12:38:47 +0100
Received: from [172.31.5.7] (helo=[172.31.5.7]) by eantc.de with ESMTP (eXpurgate 4.3.1) (envelope-from <cross@eantc.de>) id 5a9694c6-49e8-c0a864640019-ac1f0507d447-1 for <bmwg@ietf.org>; Wed, 28 Feb 2018 12:38:46 +0100
To: bmwg@ietf.org
References: <2e2f64cb-4c63-f7eb-f43b-33d9b1255cd1@eantc.de> <44a149d0-9c97-3795-4c1c-aa30a93e9a55@eantc.de>
From: Carsten Rossenhoevel <cross@eantc.de>
Message-ID: <7cb160f9-a3ef-3526-cc93-9c40c7cfddda@eantc.de>
Date: Wed, 28 Feb 2018 12:38:46 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0
MIME-Version: 1.0
In-Reply-To: <44a149d0-9c97-3795-4c1c-aa30a93e9a55@eantc.de>
Content-Type: multipart/alternative; boundary="------------F3F3B7EFA40DB93DA6E1361B"
Content-Language: en-US
X-purgate-ID: 149824::1519817926-000049E8-0BAB87EB/0/0
X-purgate-type: clean
X-purgate-size: 33695
X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de
X-purgate: This mail is considered clean (visit http://www.eleven.de for further information)
X-purgate: clean
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/AMx9PjvZcNMvcixGnnx8qYDbqvE>
Subject: [bmwg] Second draft for next-gen firewall (NGFW) performance benchmarking uploaded
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 28 Feb 2018 11:38:54 -0000

Dear BMWG Participants,

Please note that my colleague Bala Balarajah has uploaded the second 
draft of the next-generation firewall (NGFW) benchmarking methodology 
earlier this week:

https://datatracker.ietf.org/doc/draft-balarajah-bmwg-ngfw-performance/

This draft will be discussed in the interim BMWG call tomorrow. It has 
been extended with an additional test case (    Concurrent connection 
capacity), more precise text overall (22 pages now), and ideas for 
additional nine test cases.

Bala and I hope that this text finds your interest and that you might be 
inclined to review it.  We will provide a short introduction tomorrow 
and will be available for your questions. Many thanks in advance!

Best regards, Carsten


On 12/18/2017 4:35 PM, Carsten Rossenhoevel wrote:
>
> Dear Benchmarking Methodology WG,
>
> My colleague Bala Balarajah has uploaded the first draft of the 
> next-generation firewall (NGFW) benchmarking methodology for your 
> review: draft-balarajah-bmwg-ngfw-performance-00 
> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00>.
>
> Currently the document contains sections for the test setup, test bed 
> preparation, reporting guidelines and test cases (complete TOC 
> below).  Please let us know specifically:
>
> - How do you assess the first test case in section 7?  Its format and 
> level of details is meant to serve as a blueprint for additional test 
> cases.
>
> - What do you think about the test equipment configuration section, 
> specifically the traffic load profile and flows in section 4?
>
> - Section 5 is a bit unusual as it defines test bed requirements for 
> minimum performance.  These were usually taken for granted in the 
> past; for virtualized test solutions they need to be made explicit we 
> (Bala and I) feel.
>
> Any feedback and comments are very welcome!  Bala and I will process 
> them swiftly - either before Dec 22 or in the first week of January.
>
> Best regards, Carsten
>
>
>
>>     1 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-1>.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .2 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-2>
>>     2 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-2>.  Requirements  . . . . . . . . . . . . . . . . . . . . . . . .2 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-2>
>>     3 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-3>.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-3>
>>     4 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-4>.  Test Setup  . . . . . . . . . . . . . . . . . . . . . . . . .3 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-3>
>>       4.1 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-4.1>.  Testbed Configuration . . . . . . . . . . . . . . . . . .3 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-3>
>>       4.2 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-4.2>.  DUT/SUT Configuration . . . . . . . . . . . . . . . . . .4 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-4>
>>       4.3 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-4.3>.  Test Equipment Configuration  . . . . . . . . . . . . . .6 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-6>
>>         4.3.1 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-4.3.1>.  Client Configuration  . . . . . . . . . . . . . . . .7 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-7>
>>         4.3.2 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-4.3.2>.  Backend Server Configuration  . . . . . . . . . . . .8 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-8>
>>         4.3.3 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-4.3.3>.  Traffic Flow Definition . . . . . . . . . . . . . . .9 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-9>
>>         4.3.4 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-4.3.4>.  Traffic Load Profile  . . . . . . . . . . . . . . . .10 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-10>
>>     5 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-5>.  Test Bed Considerations . . . . . . . . . . . . . . . . . . .11 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-11>
>>     6 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-6>.  Reporting . . . . . . . . . . . . . . . . . . . . . . . . . .12 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-12>
>>       6.1 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-6.1>.  Key Performance Indicators  . . . . . . . . . . . . . . .13 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-13>
>>     7 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-7>.  Benchmarking Tests  . . . . . . . . . . . . . . . . . . . . .14 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-14>
>>       7.1 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-7.1>.  Throughput Performance  . . . . . . . . . . . . . . . . .15 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-15>
>>         7.1.1 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-7.1.1>.  Objective . . . . . . . . . . . . . . . . . . . . . .15 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-15>
>>         7.1.2 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-7.1.2>.  Test Setup  . . . . . . . . . . . . . . . . . . . . .15 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-15>
>>         7.1.3 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-7.1.3>.  Test Parameters . . . . . . . . . . . . . . . . . . .15 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-15>
>>         7.1.4 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-7.1.4>.  Test Procedures and expected Results  . . . . . . . .17 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-17>
>>       7.2 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-7.2>.  TCP Concurrent Connection Capacity  . . . . . . . . . . .18 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-18>
>>       7.3 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-7.3>.  TCP Connection Setup Rate . . . . . . . . . . . . . . . .18 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-18>
>>       7.4 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-7.4>.  Application Transaction Rate  . . . . . . . . . . . . . .18 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-18>
>>       7.5 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-7.5>.  SSL/TLS Handshake Rate  . . . . . . . . . . . . . . . . .18 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-18>
>>     8 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-8>.  Formal Syntax . . . . . . . . . . . . . . . . . . . . . . . .18 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-18>
>>     9 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-9>.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .18 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-18>
>>     10 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-10>. Security Considerations . . . . . . . . . . . . . . . . . . .18 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-18>
>>     11 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-11>. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .18 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-18>
>>     12 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#section-12>. Normative References  . . . . . . . . . . . . . . . . . . . .18 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-18>
>>     Appendix A 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#appendix-A>.  An Appendix  . . . . . . . . . . . . . . . . . . . .18 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-18>
>>     Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .19 
>> <https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-00#page-19>
>
> On 16.11.2017 02:54, Carsten Rossenhoevel wrote:
>>
>> Dear BMWG,
>>
>> Recently, the multi-vendor, not-for-profit NetSecOPEN 
>> <http://www.netsecopen.org> initiative has been formed to innovate 
>> network security test methodology.  The network security vendors, 
>> test equipment manufacturers and test labs involved in the initiative 
>> aim to strongly improve the applicability, reproducibility and 
>> transparency of benchmarks for next-gen firewalls (NGFW), intrusion 
>> detection/prevention systems (IDS/IPS) and unified threat management 
>> (UTM) solutions.  NetSecOPEN is chaired by Brian Monkman (Cc'ed).
>>
>> We currently develop test terminology, traffic profiles and 
>> benchmarking methodology for NGFWs to start with.  With the BMWG's 
>> consent, we would like to contribute our initial draft to BMWG and 
>> continue the standards development under this working group's 
>> guidance with the goal to create RFC(s).
>>
>> Some time next week we plan to submit the first draft for the WG's 
>> review.  Our contributions should proceed swiftly in November and 
>> December - hoping that there will be a lot to review and contribute 
>> to before the end of the year.  Any contributions are more than 
>> welcome - we really hope for peer review, contributions and 
>> innovative testing ideas from the BMWG!
>>
>> An early draft table of contents is listed below for your information 
>> (this is not a formal contribution).
>>
>> Best regards,
>>     Carsten Rossenhoevel (EANTC CTO)
>>     Balamuhunthan Balarajah (EANTC Senior Test Engineer)
>>
>> Table of Contents
>>
>> 1. Introduction
>> 2. Requirements
>> 3. Scope
>> This document is focused on test methodology for network security device benchmarking tests in term of performance metrics.  It describes the test methodology to obtain reproducible test results independently using different vendor test equipment.  By defining a full set of test configuration parameters, this document will allow users to reproduce network performance measurements and compare measurements. The benchmarking tests focus a set of key performance indicators (KPI): throughput, transaction rates, concurrent connection, connection setup rate and SSL/TLS handshake rate.
>> Devices such as firewalls, Next Generation firewalls, intrusion detection and prevention devices, application delivery controllers, deep packet inspection devices and web application firewalls generally fall into the
>> network security device category.
>>      
>> 4. Test Setup
>>      4.1. Testbed Configuration
>>      4.2. DUT/SUT Configuration
>>      4.3. Test Equipment Configuration
>>     
>> 5. Test Bed Calibration
>>      
>> 6. Reporting
>>      6.1. Testbed Software and Hardware Details
>>      6.2. Key Performance Indicators
>>
>> 7. Benchmarking Tests
>>      7.1. Throughput Performance
>>      7.2. TCP Concurrent Connection Capacity
>>      7.3. TCP Connection Setup Rate
>>      7.4. Application Transaction Rate
>>      7.5. SSL/TLS Handshake Rate
>>        
>> Appendix A. Traffic Mix Definition
>> -- 
>> Carsten Rossenhövel
>> Managing Director, EANTC AG (European Advanced Networking Test Center)
>> Salzufer 14, 10587 Berlin, Germany
>> office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
>> cross@eantc.de,http://www.eantc.de
>>
>> Place of Business/Sitz der Gesellschaft: Berlin, Germany
>> Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
>> Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
>> Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
>> EU VAT No: DE812824025
>>
>>
>> _______________________________________________
>> bmwg mailing list
>> bmwg@ietf.org
>> https://www.ietf.org/mailman/listinfo/bmwg
>
> -- 
> Carsten Rossenhövel
> Managing Director, EANTC AG (European Advanced Networking Test Center)
> Salzufer 14, 10587 Berlin, Germany
> office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
> cross@eantc.de,http://www.eantc.de
>
> Place of Business/Sitz der Gesellschaft: Berlin, Germany
> Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
> Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
> Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
> EU VAT No: DE812824025
>
>
> _______________________________________________
> bmwg mailing list
> bmwg@ietf.org
> https://www.ietf.org/mailman/listinfo/bmwg