[bmwg] Network security test methodology development at BMWG supported by NetSecOPEN initiative

Carsten Rossenhoevel <cross@eantc.de> Thu, 16 November 2017 01:54 UTC

Return-Path: <cross@eantc.de>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id 518FB1293E9 for <bmwg@ietfa.amsl.com>; Wed, 15 Nov 2017 17:54:42 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id nzQD96qezk3P for <bmwg@ietfa.amsl.com>; Wed, 15 Nov 2017 17:54:39 -0800 (PST)
Received: from obelix.eantc.de (ns.eantc.com []) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 52EC6120227 for <bmwg@ietf.org>; Wed, 15 Nov 2017 17:54:39 -0800 (PST)
Received: from ns.eantc.de ([] helo=localhost) by obelix.eantc.de with esmtp (Exim 4.80) (envelope-from <cross@eantc.de>) id 1eF9O0-0000t6-4M; Thu, 16 Nov 2017 02:54:36 +0100
Received: from [] (helo=[]) by eantc.de with ESMTP (eXpurgate 4.1.9) (envelope-from <cross@eantc.de>) id 5a0cefdb-0fa5-c0a864640019-c0a86405efd3-1 for <multiple-recipients>; Thu, 16 Nov 2017 02:54:36 +0100
To: bmwg@ietf.org
From: Carsten Rossenhoevel <cross@eantc.de>
Organization: EANTC AG
Cc: bmonkman@netsecopen.org
Message-ID: <2e2f64cb-4c63-f7eb-f43b-33d9b1255cd1@eantc.de>
Date: Thu, 16 Nov 2017 02:54:35 +0100
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="------------43E3AE1BB34A2F28F05943CF"
Content-Language: en-US
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/dl7NR-IB94-wRbBcVzeLz0cjyJw>
Subject: [bmwg] Network security test methodology development at BMWG supported by NetSecOPEN initiative
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 16 Nov 2017 01:55:36 -0000

Dear BMWG,

Recently, the multi-vendor, not-for-profit NetSecOPEN
<http://www.netsecopen.org> initiative has been formed to innovate
network security test methodology.  The network security vendors, test
equipment manufacturers and test labs involved in the initiative aim to
strongly improve the applicability, reproducibility and transparency of
benchmarks for next-gen firewalls (NGFW), intrusion detection/prevention
systems (IDS/IPS) and unified threat management (UTM) solutions. 
NetSecOPEN is chaired by Brian Monkman (Cc'ed). 

We currently develop test terminology, traffic profiles and benchmarking
methodology for NGFWs to start with.  With the BMWG's consent, we would
like to contribute our initial draft to BMWG and continue the standards
development under this working group's guidance with the goal to create

Some time next week we plan to submit the first draft for the WG's
review.  Our contributions should proceed swiftly in November and
December - hoping that there will be a lot to review and contribute to
before the end of the year.  Any contributions are more than welcome -
we really hope for peer review, contributions and innovative testing
ideas from the BMWG!

An early draft table of contents is listed below for your information
(this is not a formal contribution).

Best regards,
    Carsten Rossenhoevel (EANTC CTO)
    Balamuhunthan Balarajah (EANTC Senior Test Engineer)

Table of Contents

1. Introduction    
2. Requirements    
3. Scope
This document is focused on test methodology for network security device benchmarking tests in term of performance metrics.  It describes the test methodology to obtain reproducible test results independently using different vendor test equipment.  By defining a full set of test configuration parameters, this document will allow users to reproduce network performance measurements and compare measurements. The benchmarking tests focus a set of key performance indicators (KPI): throughput, transaction rates, concurrent connection, connection setup rate and SSL/TLS handshake rate. 
Devices such as firewalls, Next Generation firewalls, intrusion detection and prevention devices, application delivery controllers, deep packet inspection devices and web application firewalls generally fall into the
network security device category.  
4. Test Setup
    4.1. Testbed Configuration
    4.2. DUT/SUT Configuration
    4.3. Test Equipment Configuration
5. Test Bed Calibration
6. Reporting
    6.1. Testbed Software and Hardware Details
    6.2. Key Performance Indicators    

7. Benchmarking Tests   
    7.1. Throughput Performance    
    7.2. TCP Concurrent Connection Capacity    
    7.3. TCP Connection Setup Rate    
    7.4. Application Transaction Rate       
    7.5. SSL/TLS Handshake Rate    
Appendix A. Traffic Mix Definition

Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721
cross@eantc.de, http://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany
Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
EU VAT No: DE812824025