Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance
bmonkman@netsecopen.org Fri, 21 May 2021 18:37 UTC
Return-Path: <bmonkman@netsecopen.org>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id ED8D53A1ACF for <bmwg@ietfa.amsl.com>; Fri, 21 May 2021 11:37:27 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Level:
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=netsecopen-org.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YzGGsTHyA3rq for <bmwg@ietfa.amsl.com>; Fri, 21 May 2021 11:37:22 -0700 (PDT)
Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 36A583A1ACE for <bmwg@ietf.org>; Fri, 21 May 2021 11:37:21 -0700 (PDT)
Received: by mail-qt1-x82e.google.com with SMTP id h21so16007956qtu.5 for <bmwg@ietf.org>; Fri, 21 May 2021 11:37:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netsecopen-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:references:in-reply-to:subject:date:message-id :mime-version:content-transfer-encoding:thread-index :content-language; bh=l/bAaW2YSjjc3b6QruhFDAgLTTXPUhpIlXhkWMja59E=; b=LvOCdmzIfK4hpE0muEatH47OrhEI0bhlIEvwSt6SmZqs9rbEyzL51ksDqVqhcd47Ev 3ye1NC7CH//FTnKmkBQYX6gGr0TRyaEHrZN0PnvlozPGuoTJdJ6RpJDnb0W82YuRUxiI mEHXkw78mnFPh9zGMPTJN5Ui+Gk2tw8/PwQdykSobIOpNrL5RKOpcmDL775lGxKydLJ8 apGNVOXhymVxEPv5CqJ2ucBTrtCni0oDNiy5x0UMxcUiAzG1bZtHXRrDEL8CgMQTx89U Fr6XbkPbbhxpkAxxmnCbUJoKOaXhCaiw150EJ86Vf0iy+t610O5Ewn1mF6310/QEFRYb L+gQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:references:in-reply-to:subject:date :message-id:mime-version:content-transfer-encoding:thread-index :content-language; bh=l/bAaW2YSjjc3b6QruhFDAgLTTXPUhpIlXhkWMja59E=; b=ZENATqJY0GwqDt4/30nXg9Vft8ifsUfGp77ugtVMY/0SxP3a/eRdDdIN5dS06J5dgU NC6gQkW6H+xPeJBqSGc2dQq37hFqjtquRCmCWesUvKaekvWaetyFF8HqETHxW34tcjPL O2kNfE94U/oHWxvnWWWOYHq15ICV5gJ/Oo36YmDmKybWnp7dZKlOW2dX4bc0tGboSy3+ d5x9oaExEUb+U4iUB5a50fo7U0j8jPA/gADN2IbjomX0u3+LU0BqaZmN8sWnLnFCY051 I0+GEuPjK83cxKvBsW9SPlvBhUK4xm0o1bF9pSvfeEBmd4mpz1ezcL5Bdef8w4X/FI7A AYEA==
X-Gm-Message-State: AOAM531FSc8xBesU+ccch27x83Ms/nanJI+IDvZeDtZlSHZBox8LIDmc eed9qmFD4az5E3XcXAssP/jKEQ==
X-Google-Smtp-Source: ABdhPJyaiBdVwIP77m9vfLK7KXIGNL0J9zxUCGjIzxH5iT5IVUWP4DYmqDGWGDDushVKSSZkIGs9GA==
X-Received: by 2002:ac8:7215:: with SMTP id a21mr13168073qtp.306.1621622239562; Fri, 21 May 2021 11:37:19 -0700 (PDT)
Received: from DESKTOP42TMNEU ([2601:986:8001:d660:3dc5:bd9e:433:5045]) by smtp.gmail.com with ESMTPSA id h2sm5592006qkf.106.2021.05.21.11.37.18 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 May 2021 11:37:19 -0700 (PDT)
From: bmonkman@netsecopen.org
To: 'Sarah Banks' <sbanks@encrypted.net>, 'ALFRED MORTON' <acmorton@att.com>
Cc: 'Gabor LENCSE' <lencse@hit.bme.hu>, 'Bala Balarajah' <bala@netsecopen.org>, 'Bala Balarajah' <bm.balarajah@gmail.com>, bmwg@ietf.org, "'MORTON, ALFRED C (AL)'" <acm@research.att.com>
References: <413e779fd7eb4dd4b3aa8473c171e282@att.com> <f1a2b5c5-ebf2-12ab-b053-b9b2538342ad@hit.bme.hu> <047501d745bb$e22f4ab0$a68de010$@netsecopen.org> <7dc6b282-7f41-bf7c-f09c-65e7ce94b674@hit.bme.hu> <048801d745be$31424b50$93c6e1f0$@netsecopen.org> <84196d5ce7474f9196ab000be64c49fd@att.com> <02629ACE-FDA4-4ACF-9459-825521596B83@encrypted.net>
In-Reply-To: <02629ACE-FDA4-4ACF-9459-825521596B83@encrypted.net>
Date: Fri, 21 May 2021 14:37:18 -0400
Message-ID: <006c01d74e70$54718b30$fd54a190$@netsecopen.org>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Outlook 16.0
Thread-Index: AQFRbj6fJkgW8izlvCwqrrFMnkxIIgCInYIRAuwPM28Ba7bQAAJP8T6gAZobdVoCfn8Z8KufvBcQ
Content-Language: en-us
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/GJPkk-iGAqnnw3waki3tT7h5IPQ>
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 21 May 2021 18:37:28 -0000
Sarah, Thanks for your comments. Given when it was received with respect to the WGLC timeframe your suggestions will be considered for the next draft. (Which is being posted shortly.) We will get back to you in due course. Brian -----Original Message----- From: Sarah Banks <sbanks@encrypted.net> Sent: Thursday, May 20, 2021 4:35 PM To: ALFRED MORTON <acmorton@att.com> Cc: bmonkman@netsecopen.org; Gabor LENCSE <lencse@hit.bme.hu>; Bala Balarajah <bala@netsecopen.org>; Bala Balarajah <bm.balarajah@gmail.com>; bmwg@ietf.org; MORTON, ALFRED C (AL) <acm@research.att.com> Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance Hi, Sharing feedback with my participant hat on. I do apologize for sending it so late. A few high level comments: - I still land in the "I don't agree that this doc clearly covers "all next gen security devices" - and they're not clearly defined, either. For the three that are covered, I think there's a lumping of the IPS and IDS that would be better served separated. From my perspective, an IPS *is* a bump in the wire, and does make "blocking" decisions, as called out in this document. An IDS however, does not - it tells you it happened after the fact, and it's a passive device. This draft is really keyed around "active inline" functions - ie/ a bump in the wire - and I think keeping the focus there makes sense, and lumping the IDS in does more harm than good. In short, I don't support this draft moving forward with the inclusion of IDS - it doesn't make sense to me. One way forward might be to remove the IDS as a device from scope (as a suggestion). - I've been testing for a long time, but I find this draft ... uncomfortable to approach. I expected, and would have preferred, something that walks me through a specific set of recommended scenarios with specifics on what to configure for repeatable tests that I could compare results with, but there are a lot of moving parts here and a lot of assertions that have to be made for the test beds as a whole where I think the test results could vary wildly when the same topology is handed to Test Person A and Test Person B. - Most of the feedback below omits the IDS pieces, because so much of it didn't apply. Thanks, Sarah - The draft aims to replace RFC3511, but expands scope past Firewalls, to "next generation security devices". I'm not finding a definition of what a "next generation security device is", nor an exhaustive list of the devices covered in this draft. A list that includes is nice, but IMO not enough to cover what would be benchmarked here - I'd prefer to see a definition and an exhaustive list. - What is a NGIPS or NGIDS? If there are standardized definitions pointing to them is fine, otherwise, there's a lot of wiggle room here. - I still have the concern I shared at the last IETF meeting, where here, we're putting active inline security devices in the same category as passive devices. On one hand, I'm not sure I'd lump these three together in the first place; on the other, active inline devices typically include additional functions to allow administrators to control what happens to packets in the case of failure, and I don't see those test cases included here. - Section 4.1 - it reads as if ANY device in the test setup cannot contribute to network latency or throughput issues, including the DUTs - is that what you intended? - Option 1: It'd be nice to see a specific, clean, recommended test bed. There are options for multiple emulated routers. As a tester, I expect to see a specific, proscribed test bed that I should configure and test against. - Follow on: I'm curious as to the choice of emulated routers here. The previous test suggests you avoid routers and switches in the topo, but then there are emulated ones here. I'm curious as to what advantages you think these bring over the real deal, and, why they aren't subject to the same limitations previously described? - In section 4.1 the text calls out Option 1 as the preferred test bed, which includes L3 routing, but it's not clear why that's needed? - The difference between Option 1 and Option 2 is the inclusion of additional physical gear in Option 2 - it's not clear why that's needed, or why the tester can't simply directly connect the test equipment to the DUT and remove extraneous devices from potential influence on results? - Section 4.2, the table for NGFW features - I'm not sure what the difference is between RECOMMENDED and OPTIONAL? (I realize that you might be saying that RECOMMENDED is the "must have enabled" features, where as optional is at your discretion, but would suggest that you make that clear) - Proscribing a list of features that have to be enabled for the test, or at least more than 1, feels like a strange choice here - I'd have expected tests cases that either test the specific features one at a time, or suggest several combinations, but that ultimately, we'd tell the tester to document WHICH features were enabled, to make the test cases repeatable? This allows the tester to apply a same set of apples to apples configurations to different vendor gear, and omit the 1 feature that doesn't exist on a different NGFW (for example), but hold a baseline that could be tested. - Table 2: With the assumption that NGIPS/IDS are required to have the features under "recommended", I disagree with this list. For example, some customers break and inspect at the tap/agg layer of the network - in this case, the feed into the NGIDS might be decrypted, and there's no need to enable SSL inspection, for example. - Table 3: I disagree that an NGIDS IS REQUIRED to decrypt SSL. This behaviour might be suitable for an NGIPS, but the NGIDS is not a bump on the wire, and often isn't decrypting and re-encrypting the traffic. - Table 3: An NGIDS IMO is still a passive device - it wouldn't be blocking anything, but agree that it might tell you that it happened after the fact. - Table 3: Anti-evasion definition - define "mitigates". - Table 3: Web-filtering - not a function of an NGIDS. - Table 3: DLP: Not applicable for an NGIDS. - Can you expand on "disposition of all flows of traffic are logged" - what's meant here specifically, and why do they have to be logged? (Logging, particularly under high loads, will impact it's own performance marks, and colours output) - ACLs wouldn't apply to an IDS because IDS's aren't blocking traffic :) - It might be helpful to testers to say something like "look, here's one suggested set of ACLs. If you're using them, great, reference that, but otherwise, make note of the ACLs you use, and use the same ones for repeatable testing". - 4.3.1.1 The doc proscribes specific MSS values for v4/v6 with no discussion around why they're chosen - that color could be useful to the reader. - 4.3.1.1 - there's a period on the 3rd to last line "(SYN/ACL, ACK). and" that should be changed. - 4.3.1.1 - As a tester with long time experience with major test equipment manufacturers, I can't possibly begin to guess which ones of them would conform to this - or even if they'd answer these questions. How helpful is this section to the non test houses? I suggest expansion here, ideally with either covering the scope of what you expect to cover, or hopefully which (open source/generally available) test tools or emulators could be considered for use as examples. - 4.3.1.3 - Do the emulated web browser attributes really apply to testing the NGIPS? - 4.3.2.3 - Do you expect to also leverage TLS 1.3 as a configuration option here? - 4.3.4 - I'm surprised to see the requirement that all sessions establish a distinct phase before moving on to the next. You might clarify why this is a requirement, and why staggering them is specifically rejected? - 5.1 - I like the sentence, but it leaves a world of possibilities open as to how one confirmed that the ancillary switching or routing functions didn't limit the performance, particularly the virtualized components? - 5.3 - this is a nice assertion but again, how do I reasonably make the assertion? - 6.1 - I would suggest that the test report include the configuration of ancillary devices on both client/server side as well - 6.3 - Nothing on drops anywhere? - 7.1.3.2 - Where are these numbers coming from? How are you determining the "initial inspected throughput"? Maybe I missed that in the document overall, but it's not clear to me where these KPIs are collected? I suggest this be called out. - 7.1.3.3 - what is a "relevant application traffic mix" profile? - 7.1.3.4 - where does this monitoring occur? - 7.1.3.4 - This looks a bit like conformance testing - Why does item (b) require a specific number/threshold? - 9: Why is the cipher squite recommendation for a real deployment outside the scope of this document?
- [bmwg] FW: WGLC on New version of draft-ietf-bmwg… MORTON JR., AL
- Re: [bmwg] FW: WGLC on New version of draft-ietf-… Gabor LENCSE
- Re: [bmwg] FW: WGLC on New version of draft-ietf-… bmonkman
- Re: [bmwg] FW: WGLC on New version of draft-ietf-… Gabor LENCSE
- Re: [bmwg] FW: WGLC on New version of draft-ietf-… bmonkman
- Re: [bmwg] FW: WGLC on New version of draft-ietf-… MORTON JR., AL
- [bmwg] Second part -- Re: FW: WGLC on New version… Gabor LENCSE
- Re: [bmwg] Second part -- Re: FW: WGLC on New ver… bmonkman
- [bmwg] Sequential vs. random -- Re: FW: WGLC on N… Gábor LENCSE
- Re: [bmwg] FW: WGLC on New version of draft-ietf-… MORTON JR., AL
- Re: [bmwg] FW: WGLC on New version of draft-ietf-… bmonkman
- Re: [bmwg] Sequential vs. random -- Re: FW: WGLC … bmonkman
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… Sarah Banks
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… bmonkman
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… bmonkman
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… bmonkman
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… bmonkman
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… Sarah Banks
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… Sarah Banks
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… bmonkman
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… MORTON JR., AL
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… bmonkman
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… Carsten Rossenhoevel
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… MORTON JR., AL
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… Sarah Banks
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… bmonkman
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… Carsten Rossenhoevel
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… Sarah Banks
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… bmonkman
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… MORTON JR., AL
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… bmonkman
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… Gábor LENCSE
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… Carsten Rossenhoevel
- Re: [bmwg] WGLC on New version of draft-ietf-bmwg… Gábor LENCSE