Re: [bmwg] Network security test methodology development at BMWG supported by NetSecOPEN initiative

"MORTON, ALFRED C (AL)" <> Thu, 16 November 2017 03:58 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 7D5DA12943C for <>; Wed, 15 Nov 2017 19:58:23 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -5.39
X-Spam-Status: No, score=-5.39 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-2.8, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01] autolearn=ham autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id cjTbqFCFScGk for <>; Wed, 15 Nov 2017 19:58:20 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 3F543127010 for <>; Wed, 15 Nov 2017 19:58:20 -0800 (PST)
Received: from pps.filterd ( []) by ( with SMTP id vAG3tbKr021363; Wed, 15 Nov 2017 22:58:16 -0500
Received: from ( []) by with ESMTP id 2e92b6gupd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 15 Nov 2017 22:58:16 -0500
Received: from (localhost []) by (8.14.5/8.14.5) with ESMTP id vAG3wEaF122160; Wed, 15 Nov 2017 21:58:15 -0600
Received: from ( []) by (8.14.5/8.14.5) with ESMTP id vAG3wBlE122117 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 15 Nov 2017 21:58:12 -0600
Received: from ( []) by (RSA Interceptor); Thu, 16 Nov 2017 03:57:55 GMT
Received: from (localhost []) by (8.14.5/8.14.5) with ESMTP id vAG3vt4K013145; Wed, 15 Nov 2017 21:57:55 -0600
Received: from ( []) by (8.14.5/8.14.5) with ESMTP id vAG3voqS012886; Wed, 15 Nov 2017 21:57:50 -0600
Received: from ( []) by (Postfix) with ESMTP id 3D863E3B79; Wed, 15 Nov 2017 22:57:49 -0500 (EST)
Received: from ([fe80::b09c:ff13:4487:78b6]) by ([fe80::d550:ec84:f872:cad9%15]) with mapi id 14.03.0361.001; Wed, 15 Nov 2017 22:57:49 -0500
From: "MORTON, ALFRED C (AL)" <>
To: Carsten Rossenhoevel <>, "" <>
CC: "" <>
Thread-Topic: [bmwg] Network security test methodology development at BMWG supported by NetSecOPEN initiative
Thread-Index: AQHTXn4HQuc7NQ5jKEyY1xeR259OI6MWXOyQ
Date: Thu, 16 Nov 2017 03:57:48 +0000
Message-ID: <>
References: <>
In-Reply-To: <>
Accept-Language: en-US
Content-Language: en-US
x-originating-ip: []
Content-Type: multipart/alternative; boundary="_000_4D7F4AD313D3FC43A053B309F97543CF49046341njmtexg5researc_"
MIME-Version: 1.0
X-RSA-Inspected: yes
X-RSA-Classifications: public, General SSNFP Patterns II
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-11-16_01:, , signatures=0
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1711160053
Archived-At: <>
Subject: Re: [bmwg] Network security test methodology development at BMWG supported by NetSecOPEN initiative
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 16 Nov 2017 03:58:23 -0000

Hi Carsten and Bala,

Thanks for your contribution and proposal on our mailing list.

It’s fairly clear that your initial proposal to benchmark
Next-Gen Firewalls is within our current (and about to be
revised) WG Charter, since closely related work was completed
in 2003, in RFC 3511 (for Firewalls).

It will be interesting to see how benchmarking might be
conducted for intrusion detection/prevention systems (IDS/IPS)
and unified threat management (UTM) solutions, and how
interactions with the IETF Security Area that could help.
But we will cross that bridge when we come to it.

So, the contribution-driven process here at IETF doesn’t
require WG consent, but we want everyone to become familiar with
IETF’s IPR policy
and other aspects of IETF participation
as you join us.

We also have a “getting started” page for BMWG here:

Looking forward to continued interactions.

bmwg co-chair

From: bmwg [] On Behalf Of Carsten Rossenhoevel
Sent: Wednesday, November 15, 2017 8:55 PM
Subject: [bmwg] Network security test methodology development at BMWG supported by NetSecOPEN initiative

Dear BMWG,

Recently, the multi-vendor, not-for-profit NetSecOPEN<> initiative has been formed to innovate network security test methodology.  The network security vendors, test equipment manufacturers and test labs involved in the initiative aim to strongly improve the applicability, reproducibility and transparency of benchmarks for next-gen firewalls (NGFW), intrusion detection/prevention systems (IDS/IPS) and unified threat management (UTM) solutions.  NetSecOPEN is chaired by Brian Monkman (Cc'ed).

We currently develop test terminology, traffic profiles and benchmarking methodology for NGFWs to start with.  With the BMWG's consent, we would like to contribute our initial draft to BMWG and continue the standards development under this working group's guidance with the goal to create RFC(s).

Some time next week we plan to submit the first draft for the WG's review.  Our contributions should proceed swiftly in November and December - hoping that there will be a lot to review and contribute to before the end of the year.  Any contributions are more than welcome - we really hope for peer review, contributions and innovative testing ideas from the BMWG!

An early draft table of contents is listed below for your information (this is not a formal contribution).

Best regards,
    Carsten Rossenhoevel (EANTC CTO)
    Balamuhunthan Balarajah (EANTC Senior Test Engineer)

Table of Contents

1. Introduction

2. Requirements

3. Scope

This document is focused on test methodology for network security device benchmarking tests in term of performance metrics.  It describes the test methodology to obtain reproducible test results independently using different vendor test equipment.  By defining a full set of test configuration parameters, this document will allow users to reproduce network performance measurements and compare measurements. The benchmarking tests focus a set of key performance indicators (KPI): throughput, transaction rates, concurrent connection, connection setup rate and SSL/TLS handshake rate.

Devices such as firewalls, Next Generation firewalls, intrusion detection and prevention devices, application delivery controllers, deep packet inspection devices and web application firewalls generally fall into the

network security device category.

4. Test Setup

    4.1. Testbed Configuration

    4.2. DUT/SUT Configuration

    4.3. Test Equipment Configuration

5. Test Bed Calibration

6. Reporting

    6.1. Testbed Software and Hardware Details

    6.2. Key Performance Indicators

7. Benchmarking Tests

    7.1. Throughput Performance

    7.2. TCP Concurrent Connection Capacity

    7.3. TCP Connection Setup Rate

    7.4. Application Transaction Rate

    7.5. SSL/TLS Handshake Rate

Appendix A. Traffic Mix Definition


Carsten Rossenhövel

Managing Director, EANTC AG (European Advanced Networking Test Center)

Salzufer 14, 10587 Berlin, Germany

office +49.30.3180595-21, fax +49.30.3180595-10, mobile +49.177.2505721<>,<>

Place of Business/Sitz der Gesellschaft: Berlin, Germany

Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus

Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk

Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany

EU VAT No: DE812824025