Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
bmonkman@netsecopen.org Wed, 23 December 2020 15:35 UTC
Return-Path: <bmonkman@netsecopen.org>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B1C183A0AFA for <bmwg@ietfa.amsl.com>; Wed, 23 Dec 2020 07:35:47 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Level:
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=netsecopen-org.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wvTNLdnxH_9M for <bmwg@ietfa.amsl.com>; Wed, 23 Dec 2020 07:35:45 -0800 (PST)
Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D7A273A0AEE for <bmwg@ietf.org>; Wed, 23 Dec 2020 07:35:44 -0800 (PST)
Received: by mail-qk1-x735.google.com with SMTP id 143so15199677qke.10 for <bmwg@ietf.org>; Wed, 23 Dec 2020 07:35:44 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netsecopen-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:references:in-reply-to:subject:date:message-id :mime-version:content-transfer-encoding:thread-index :content-language; bh=s0vfGK185TEiXLIy/PHT/dAax4jNv96J8LtWwZ7Owmg=; b=M4xMn+Ed+wwsDszxmghjVY/l+d0AxwquClDuXSWGhBL9bGy0+UIHJQTMco5Dk8s3cU x4zAAA9F6ZGhrfeEhTvheDFaqfDHRLC60E+8nMcBiw0FAiJ3BH975fFl43M6bgcpAzFE XwLVB6mfqGJsQvgSur601dvI3X1+dI3rWZCfukuWq0panfP0x5/BOG3TEWefvpxgsP7j IbTPGZBJkzWbKqDqztoils3oV/tEIIAyJszcJJ9RYbVNV8EQn95XbPBhPqJJI2WywHHE AWWHtPbP9gex66+c9WLfgfVDPDWn75SgTAge5TAqRZJJVI5Wvlag2ad8NOrtdht9QLhz 6rRg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:references:in-reply-to:subject:date :message-id:mime-version:content-transfer-encoding:thread-index :content-language; bh=s0vfGK185TEiXLIy/PHT/dAax4jNv96J8LtWwZ7Owmg=; b=e3uXBsqhNh0g+bHkQzBkO1xCHqUNDn/qzpLIxg2NT1fLsO1QxvNiYXP2TA9THP3YKa lWU4nCjrNHmJm+9EpuOgoNQmcN4tkM04FuKe+DFnp/fPLHTX9rEexPL0T0pKkcZPNOID WljdhpwX3YiBGSq7ul3jdfbJba0eInKIPXXRslHGIUGXLY01jdg7D8oRA6+EopjOwrBz pUn/c8PTIMwPk5h7rjOYGgvh2G9iqupbdtAwqjVOilB6R7EqjEvxv0G3lAVM0YBu8uDC HdCYEYSxf2Vg186FX0webXso0OITmh6nVJcYfjuo10YRh2ynB/9YbhpXVkX1vsveQQcg S+ug==
X-Gm-Message-State: AOAM532A7+t69+bjLD23+aZ29bd1F90GiXkSQvy/8A3jCk6mM4lagRy5 I50qZAwIVva/QgOfjaTCGTDUuQTnEE0k/lz5
X-Google-Smtp-Source: ABdhPJz3q4hcv+XTaCIe3FvuEKgD+i1B6LvdIu6d6Z1KM4lyDKrQ0YMEXWeF2wQm+CvI6Ku+mYcGfQ==
X-Received: by 2002:a37:2c82:: with SMTP id s124mr26932429qkh.130.1608737743588; Wed, 23 Dec 2020 07:35:43 -0800 (PST)
Received: from WINDOWSU6SOVGL (c-98-235-212-118.hsd1.pa.comcast.net. [98.235.212.118]) by smtp.gmail.com with ESMTPSA id c139sm13711854qke.24.2020.12.23.07.35.42 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Dec 2020 07:35:42 -0800 (PST)
From: bmonkman@netsecopen.org
To: "'Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)'" <vrpolak=40cisco.com@dmarc.ietf.org>, bmwg@ietf.org
Cc: "'MORTON, ALFRED C (AL)'" <acm@research.att.com>
References: <4D7F4AD313D3FC43A053B309F97543CF014767877B@njmtexg5.research.att.com> <BY5PR11MB4038A9A8D3A833F8AA2B03A7BDDE0@BY5PR11MB4038.namprd11.prod.outlook.com>
In-Reply-To: <BY5PR11MB4038A9A8D3A833F8AA2B03A7BDDE0@BY5PR11MB4038.namprd11.prod.outlook.com>
Date: Wed, 23 Dec 2020 10:35:42 -0500
Message-ID: <01ff01d6d941$4629bb50$d27d31f0$@netsecopen.org>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Outlook 16.0
Thread-Index: AQEuqiYZGMRTjsr9V6cISPU4hr9IhQFfZTXVq0oytYA=
Content-Language: en-ca
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/XpkQs-gWFfKMLhacSK3rJCcInKk>
Subject: Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 23 Dec 2020 15:35:48 -0000
Al, I think I can speak on behalf of the other authors and state we are willing to take the necessary time to review Vratko's input and create another version of the draft. Brian -----Original Message----- From: bmwg <bmwg-bounces@ietf.org> On Behalf Of Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco) Sent: December 23, 2020 10:31 AM To: bmwg@ietf.org Cc: MORTON, ALFRED C (AL) <acm@research.att.com> Subject: Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05 > Please read and express your opinion on whether or not this > Internet-Draft should be forwarded to the Area Directors for > publication as an Informational RFC. The current draft is a large document, and I will have multiple comments. I expect some of them will be addressed by creating -06 version, so my opinion is -05 should not be forwarded for publication. > Send your comments to this list or to co-chairs at > bmwg-chairs@ietf.com The issue is, I do not have all the comments ready yet. In general, I need to spend some effort when turning my nebulous ideas into coherent sentences (mostly because only when writing the sentences I realize the topic is even more complicated than I thought at first). Also, specifically for BMWG, I want my comments to be more complete than usual. Not just "I do not like/understand this sentence", but give a new sentence and a short explanation why the new sentence is better. I have two reasons for aiming for high quality comments. First, I imagine many people are reading this list. That means, if I write a lazy superficial comment, I save my time, but readers will spend more time trying to reconstruct my meaning. (Similar to how in software development, code is written once but read many times.) Second reason is high latency on this mailing list. Usually, by the time the author reacts to the comments, the reviewer has switched their attention to other tasks, so it is better when the first comment does not need any subsequent clarifications from the reviewer. > allow for holidays and other competing topics I reserved some time before holidays, originally for improving MLRsearch, but NGFW is closer to publishing so it takes precedence. My plan is to start with giving a few low-quality comments, mainly to hint what areas I want to see improved. After holidays, I will write higher quality comments, one e-mail per area. This e-mail contains the low-quality comments (in decreasing order of brevity). 1. Test Bed Considerations. Useful, but maybe should be expanded into a separate draft. (Mainly expanding on "testbed reference pre-tests", and what to do if they fail but we still want some results.) 2. Sentence with "safety margin of 10%". Unclear. If you want to add or subtract, name both the quantity before and after the operation, so in later references it is clear which quantity is referenced. Also, why 10% and not something else (e.g. 5%)? 3. Is it "test bed" or "testbed"? I assume it means "SUT" plus "test equipment" together, but is should be clarified. 4. Sustain phase follows after ramp-up phase immediately, without any pause, right? Then there is in-flight traffic at sustain phase start and end, making it hard to get precise counters. 5. Validation criteria. The draft contains terms "target throughput" and "initial throughput", but also phrases like "the maximum and average achievable throughput within the validation criteria". I am not even sure if validation criteria apply to a trial (e.g. telemetry suggests test equipment behavior was not stable enough) or a whole search (e.g. maximum achievable throughput is below acceptance threshold). 6. It seems the same word "throughput" is used to mean different quantities depending on context. Close examination suggests it probably means forwarding rate [0] except the offered load [1] is not given explicitly (and maybe is not even constant). When I see "throughput" I think [2] (max offered load with no loss), which does not work as generally the draft allows some loss. Also, some terms (e.g. "http throughput") do not refer to packets, but other "transactions". 7. SUT state affecting performance. The draft does not mention any, so I think it assumes "stateless" SUT. An example of "stateful" SUT is NAT, where opening sessions has smaller performance than forwarding on already opened sessions. Or maybe it is assumed any such state enters a stationary state during ramp-up, so in sustain phase the performance is stable (e.g. NAT sessions may be timing out, but in a stable rate). 8. Stateless or stateful traffic generation. Here stateless means predetermined packets are sent at predetermined times. Stateful means time or content of next-to-send packet depends on time or content of previously received packets. Draft section 7.1 looks like stateless traffic to me (think IMIX [3]), while others look like stateful (you cannot count http transaction rate from lossy stateless traffic). In general, stateful traffic is more resource intensive for test equipment, so it is harder to achieve high enough offered load. Also, stateful traffic generation is more sensitive to packet loss and latency of SUT. Vratko. [0] https://tools.ietf.org/html/rfc2285#section-3.6.1 [1] https://tools.ietf.org/html/rfc2285#section-3.5.2 [2] https://tools.ietf.org/html/rfc2544#section-26.1 [3] https://tools.ietf.org/html/rfc6985 -----Original Message----- From: bmwg <bmwg-bounces@ietf.org> On Behalf Of MORTON, ALFRED C (AL) Sent: Friday, 2020-December-18 19:16 To: bmwg@ietf.org Subject: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-performance-05 Hi BMWG, We will start a WG Last Call for Benchmarking Methodology for Network Security Device Performance https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-05 The WGLC will close on 22 January, 2021, allow for holidays and other competing topics (IOW, plenty of time!) Please read and express your opinion on whether or not this Internet-Draft should be forwarded to the Area Directors for publication as an Informational RFC. Send your comments to this list or to co-chairs at bmwg-chairs@ietf.com for the co-chairs, Al _______________________________________________ bmwg mailing list bmwg@ietf.org https://www.ietf.org/mailman/listinfo/bmwg _______________________________________________ bmwg mailing list bmwg@ietf.org https://www.ietf.org/mailman/listinfo/bmwg
- [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-perform… MORTON, ALFRED C (AL)
- Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-per… Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)
- Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-per… bmonkman
- Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-per… bmonkman
- Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-per… bmonkman
- Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-per… MORTON, ALFRED C (AL)
- Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-per… bmonkman
- Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-per… MORTON, ALFRED C (AL)
- Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-per… Vratko Polak -X (vrpolak - PANTHEON TECH SRO at Cisco)
- Re: [bmwg] WG Last Call: draft-ietf-bmwg-ngfw-per… Maciek Konstantynowicz (mkonstan)