Re: [bmwg] Tsvart last call review of draft-ietf-bmwg-ngfw-performance-12

[Carsten Rossenhoevel <[e-mail]>]

Hi Tommy and BMWG Subscribers,

Happy New Year!  Apologies for the late response due to the Christmas 
break.

Tommy, thank you for your review of the next-generation firewall 
benchmarking draft and for your comments submitted on Dec 17.

In testing, it is important to document any tiny detail of the test 
setup and environment.  The goal is to ensure reproducibility and 
comparability of results.  An under-specification of details was one of 
the issues with RFC3511.  Configurable TCP parameters can highly 
influence benchmarking test results.  With the goal to achieve 
reproducible test results we decided to specify TCP parameter values, by 
the way aligned with defaults of most of today's operating systems.

Using normative language is the only way from the editors' point of view 
to achieve a level-playing field for all users (vendors, network 
operators, end users) to always achieve comparable results when testing 
the same system/device.  In the whole review (more than 20 drafts over 
three years in total), the normative language was extensively reviewed 
and found adequate.

Ensuring that the test bed is up to the task (enabling performance 
benchmarking of a firewall) and not hampered by its own limitations is 
another important step (section 4.1).  The editors confirm that it is 
normatively required for the test bed to be adequate for the performance 
tests.  If you find that sub-par test beds//with inherent issues in 
throughput and latency /should/ be permitted, please explain why.

Thank you for noticing the incorrect long name for QUIC in section 
4.3.1.3.  We were unaware that the long name has been omitted by IETF 
policy.  Thus, we would edit the respective sentence and remove the long 
name.  Furthermore, could you please point us to the RFC specifying that 
HTTP/3 is by definition equal to HTTP over QUIC?  There was a 2018 
informal IETF statement, but we could not find this mentioned as 
normative language in RFC9000.

We will make an editorial change for clarification, adding HTTP/3 to the 
server side (section 4.3.2.3).

The benchmarking methodology could be complemented with synthetic 
loss/delay, but this is out of scope of this draft.  In fact, adding 
synthetic frame loss in TCP test environments often creates results that 
are quite cumbersome to interpret.  We would not recommend it.

The choice of TCP as a transport protocol has been made deliberately, 
representing what the editors and contributors deemed realistic test 
methodology for NGFWs.  Contributions for other transports are welcome 
in BMWG I would guess; since this would require another 20-30 pages from 
the point of view of the editors, probably a fresh start with a new 
draft, complementing the upcoming RFC would suit the purpose best.

Regarding the use of "throughput" in section 7.7 and elsewhere, it is 
defined as "Inspected Throughput" in section 6.3. Throughput is defined 
on layer 2 because that is the overwhelming informal understanding when 
qualifying a switch, router, or firewall.  We had quite extensive 
discussions on this topic; you may want to consult with the IETF mailing 
list archive and WG session recordings.

Best regards, Carsten


Am 12/17/2021 um 6:18 PM schrieb Tommy Pauly via Datatracker:
> Reviewer: Tommy Pauly
> Review result: On the Right Track
>
> This document has been reviewed as part of the transport area review team's
> ongoing effort to review key IETF documents. These comments were written
> primarily for the transport area directors, but are copied to the document's
> authors and WG to allow them to address any issues raised and also to the IETF
> discussion list for information.
>
> When done at the time of IETF Last Call, the authors should consider this
> review as part of the last-call comments they receive. Please always CC
> [e-mail]  if you reply to or forward this review.
>
> This document is a useful update to previous work, and it’s good to see this
> methodology being refreshed. Thanks to the authors and the WG for their work on
> this!
>
> >From a transport perspective, I’m concerned that some things are over-specified
> (details of TCP implementations) and others are underspecified (how throughput
> is measured, how loss and delay are tested). There are some nods towards
> non-TCP transports (UDP port configuration and HTTP/3 tests), but these are
> inconsistent. I’d like to see transports (TCP/UDP/QUIC/other) be treated more
> consistently throughout the document, particularly since non-TCP traffic will
> become increasingly relevant for the devices these tests are targeting.
>
> Section 4.1 says: “Testbed configuration MUST ensure that any performance
> implications that are discovered during the benchmark testing aren't due to the
> inherent physical network limitations such as the number of physical links and
> forwarding performance capabilities (throughput and latency) of the network
> devices in the testbed.”
>
> This seems like a hard thing to make normative. It’s not about
> interoperability, but rather talking about the necessary conditions to get
> useful measurements. I’d suggest a non-normative phrasing that explains the
> consequence of not checking this.
>
> Overall, I suggest reviewing your use of normative language, and prefer to use
> non-normative language when you are just describing facts rather than necessary
> actions on the behalf on an implementation or test setup.
>
> Also in section 4.1: can (DUT/SUT) be defined in the sentence earlier, when the
> terms are first introduced?
>
> It would be good to explain/define “fail-open” behavior. I assume I know what
> you mean, but there can be various types of failing open.
>
> In the criteria to match in Figure 3, the Transport layer discusses filtering
> on TCP/UDP ports, and the IP layer discusses filtering on addresses. There is
> no mention, however, of how alternate transport IP protocols (like SCTP, not
> TCP or UDP) would be filtered, or how such a security device may have separate
> rules for different transport protocols that run over UDP (like QUIC and
> SCTP-over-UDP) could be treated. This may not be standard practice, but it may
> be interesting and useful to point to.
>
> The client configuration section 4.3.1.1 details TCP stack configuration, but
> does not address other transports. Discussing QUIC seems like it will be
> relevant soon.
>
> Overall, for this section, I am struck that there’s a lot of detail that seems
> over-specified, with lots of normative language. For example, the TCP
> connection MUST end with a three- or four-way handshake. What if there’s a RST?
> I don’t understand what we’re requiring of these TCP implementations apart from
> being a functional and compliant TCP implementation. How much of this is
> actually required? Also, RFC 793 is not provided as a link, but is just a text
> reference.
>
> In Section 4.3.1.3, there are problems with the references to QUIC. QUIC is not
> an acronym for “Quick UDP Internet Connections”; that should be removed. Also,
> the text contrasts HTTP/2 and QUIC. If you are comparing to HTTP/3, please
> reference the draft (and soon-to-be RFC) for HTTP/3, not QUIC. It also doesn’t
> make sense to say “if you are testing HTTP/3, you MUST use QUIC”. HTTP/3 is
> definitionally HTTP over QUIC, so this normative requirement is
> unnecessary—just as it would be unnecessary to say that HTTP/2 MUST run over
> TCP.
>
> The client sections mentions HTTP/3, but that is not mentioned for the server
> (Section 4.3.2.3).
>
> Should the tests include results when including synthetic loss or delay to
> better emulate realistic network conditions? I understand many of the
> recommendations to remove uncontrolled delay and loss (in section 5), but
> various transport properties will change with loss and delay (and congestion,
> and buffering), which will influence performance of the DUT. Ideally, these
> DUTs would perform well in those conditions as well.
>
> Section 6.3 is again quite TCP-centric, and should be expanded for other
> transports.
>
> For metrics like “HTTP throughput” (section 7.7), how is throughput being
> rigorously defined? Is this measuring TCP bytes, TLS stream bytes, or actual
> application payload bytes?
>
>
-- 
Carsten Rossenhövel
Managing Director, EANTC AG (European Advanced Networking Test Center)
Salzufer 14, 10587 Berlin, Germany
office +49.30.3180595-21, fax [phone], mobile [phone]
[e-mail],https://www.eantc.de

Place of Business/Sitz der Gesellschaft: Berlin, Germany
Chairman/Vorsitzender des Aufsichtsrats: Herbert Almus
Managing Directors/Vorstand: Carsten Rossenhövel, Gabriele Schrenk
Registered: HRB 73694, Amtsgericht Charlottenburg, Berlin, Germany
EU VAT No: DE812824025