Re: [bmwg] draft-lencse-bmwg-benchmarking-stateful

[Gábor LENCSE <[e-mail]>]

Dear Sándor,

Thank you very much for reading and commenting our draft!

Please see my answers inline.

2021.06.21. 23:01 keltezéssel, Sandor R. Repas Dr. írta:
>
> Dear Gabor,
>
> I have read your draft.
>
> In my opinion, a good standard procedure for measuring NATxy boxes is 
> very useful and important.
>
> I assume a very important problem is NAT protocol helpers (e.g. ftp, 
> pptp, tftp). Their implementations on the NATxy boxes can greatly 
> affect the performance of the device. So it is important to take these 
> into account. A possible solution is to omit well-known ports of these 
> protocols from the target ports, or to consider some statistical 
> distributions of them in the measurement process. (And I have more 
> ideas...) I suggest a deeper investigation of this problem and 
> possible modifications based on the results.
>

I agree with you that NAT protocol helpers may have a significant impact 
on the throughput of the NATxy boxes and, therefore, the distribution of 
the destination port numbers matters a lot.

On the one hand Section 4.5 of RFC 4814 (the authors of which I have 
CC-d), requires a uniform distribution of the destination port numbers 
in the range of [1, 49151].
On the other hand, I think that the Author of RFC 4814 did not intend to 
say anything about the popularity of the different protocols. They 
focused on the hashing effect of the port numbers, that is, how the test 
frames are distributed among the different CPU-s of the DUT.

Authors, did I understand your intention well?

If it is so, then I can imagine two major approaches to handle the 
problem of NAT protocol helpers:

Approach 1.

Collect statistic about the popularity of the different protocols (that 
is the distribution of the destination port numbers) and generate 
pseudorandom destination port numbers by following that distribution. 
Hopefully, pseudorandom source port numbers will still do an appropriate 
hashing.
In fact, our draft has already done a similar thing by limiting the 
destination port numbers to avoid a DoS attack against the connection 
tracking table of the DUT, please refer to Section 4. 1 
https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-stateful#section-4.1 

However, these two solutions may conflict with each other, therefore, I 
would prefer the following approach.

Approach 2.

Completely eliminate the phenomenon of protocol helpers (either by 
disabling them or leaving out their port numbers, perhaps by leaving out 
all the system port for simplicity). We recommended to use a "narrow" 
range for destination port numbers. So this range should not overlap 
with the range of the system ports.
And apply and additional benchmarking test according to "Benchmarking 
Methodology for Network Security Device Performance", the (the authors 
of which I have also CC-d) the latest version of which is available 
from: 
https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-ngfw-performance-09

Authors of that draft, do you think that Approach 2 can be feasible?

> I have found some nits:
>
> On page 3: the stateful DUT would -> the stateful device under test 
> (DUT) would
>
> On page 4: it can use any source and destination port numbers -> it 
> can use any source and destination port numbers (from 1-65535)
>
> On page 5: for source and destination port numbers to avoid the denial 
> of service attack against the connection tracking table -> for source 
> and destination port numbers to avoid the denial of service attack 
> like event against the connection tracking table
>
> On page 7: Important warning: in normal router testing, the port 
> number -> Important warning: in normal (non-NAT) router testing, the 
> port number
>
> On page 10: results in a frame rate that is too low to keep alive the 
> connection tracking table entries -> results in a frame rate that is 
> too low to prevent the deletion of the connection tracking table 
> entries of the DUT due to timeout
>
> On page 10: the refreshing of the corresponding connection tracking 
> table element -> the refreshing of the corresponding connection 
> tracking table element of the DUT
>

Thank you very much for pointing them out!

We will address them in the next version.

Best regards,

Gábor
>
> Best regards,
>
> Sandor R Repas
>
> *From:*bmwg <[e-mail]> *On Behalf Of *Gábor LENCSE
> *Sent:* 2021. június 9., szerda 11:21
> *To:* [e-mail]
> *Cc:* Marius Georgescu <[e-mail]>
> *Subject:* Re: [bmwg] draft-lencse-bmwg-benchmarking-stateful
>
> Dear Łukasz,
>
> First of all, thank you very much for your comments!
>
> Let me add something to the answer of my co-author.
>
> Regarding the TCP-specific performance of the devices, Section 8 of 
> RFC 8219 has recommended two tests: Concurrent TCP Connection Capacity 
> and Maximum TCP Connection Establishment Rate, the measurement 
> procedures were taken verbatim from RFC 3511. (Please see: 
> https://datatracker.ietf.org/doc/html/rfc8219#section-8 
> <https://datatracker.ietf.org/doc/html/rfc8219#section-8>)
>
> So I believe that we can rely on the methods recommended in Section 
> 5.2 and 5.3 of RFC 3511. They seem to be fairly reliable to me: they 
> validate the TCP connections by transferring some small objects using 
> HTTP. (I have also cc-d Marius Georgescu, the first author of RFC 
> 8219, who can confirm it. My role in RFC 8219 was limited to Section 9 
> about DNS64 benchmarking.)
>
> In our current draft, we try to focus on the packet forwarding 
> performance of NATxy gateways. Here, it is essential that we can send 
> test frames (containing IP packet, which are containing either TCP 
> segments or UDP datagrams) having any (standard) frame sizes at any 
> frame rate without the restrictions of the TCP flow control / 
> congestion control algorithms, otherwise it seems to be impossible to 
> me to perform the Throughput test, which is also the basis for other 
> tests like the Latency test.
>
> I HOPE, but I do no know (because I have not checked it) that the 
> stateful NATxy implementations handle TCP and UDP in a uniform way, as 
> they need only the source and destination port numbers from the 
> TCP/UCP header for their stateful NATxy operation.
>
> Do you think that we should check this hypothesis?
>
> Best regards,
>
> Gábor
>
> 2021.06.03. 6:37 keltezéssel, Keiichi SHIMA írta:
>
>       Łukasz
>
>         However, until that, TCP stays major protocol used by essentially everything, including - recently to bigger degree - DNS (DNSSEC, DoT & DoH).
>
>     Yes, I 100% agree that TCP remains for a long time.
>
>         As for the concerns on testing - we can introduce concept of „ramp up” during test, which is to lessen the impact of blasting DUT with maximum amount of flows at the same time, and instead ramp them up from baseline of X, with Y connections per second within next Z seconds (for example - 60 seconds). Of course, we could potentially recommend doing *both* of the tests, to show peak and nominal performance, so the lab test team would be able to asses results in wider spectrum of operating regimes.
>
>     It seems a good approach. I guess, there already exist a few (or more?) implementations to perform that type of testing for TCP (some kind of load-testing software or whatever). I am not saying we don’t need to consider TCP in draft-lencse-bmwg-benchmarking-stateful even though there are some load-testing suites. But I believe we need to consider and decide what index of stateful NATxy boxes we want to measure in the context of the draft.
>
>     The point of draft-lencse-bmwg-benchmarking-stateful is to extend the idea of RFC2544 and RFC8219 to statefull NATxy boxes. Although RFC2544 assumes that the testing with UDP, I agree that considering TCP connection handling is important here, since stateful NATxy boxes are not simple IP forwarders, but they are kind of L4 devices as I said in my previous mail.
>
>     The state management issue happens also with UDP when using stataful NATxy boxes. draft-lencse-bmwg-benchmarking-stateful already has mentioned about the issue. It may be a good idea to extend that part to support TCP.
>
>     About the forwarding performance measurement, I am not sure if we need a special care for TCP, since looking up state table and performing address translation is the same for both TCP/UDP… If they are implemented differently, yes, maybe it need to be tested separately. In that case, the tester box must implement TCP stack and need to handle a tons of connections.
>
>     Regards,
>
>     ---
>
>     Keiichi SHIMA (島  慶一)
>
>     WIDE project<[e-mail]>  <mailto:[e-mail]>
>
>        PGP: 9D95 8544 A5CE D530 9230  DF1C BB6E ABE1 D91F EDFC
>
>     Research Laboratory, IIJ Innovation Institute, Inc<[e-mail]>  <mailto:[e-mail]>
>
>        PGP: 1B15 ACBC 1FB8 7325 B9BF  DCE6 B74D 1CFB 5173 3681
>
>         On Jun 1, 2021, at 15:53, Łukasz Bromirski<[e-mail]>  <mailto:[e-mail]>  wrote:
>
>         Keiichi,
>
>         There are two things we need to consider here.
>
>         First of all, with recent QUIC RFC, world will move slowly but surely to UDP as part of HTTP/3. We won’t see this happening massively this year, but so won’t IPv6 adoption.
>
>         However, until that, TCP stays major protocol used by essentially everything, including - recently to bigger degree - DNS (DNSSEC, DoT & DoH).
>
>         The issue with any middle box is that vendors typically optimize processing for specific protocol. While measuring performance for UDP is great for gaming and voice (typically), TCP brings additional processing overhead which will make the box (typically) perform slower.
>
>         As for the concerns on testing - we can introduce concept of „ramp up” during test, which is to lessen the impact of blasting DUT with maximum amount of flows at the same time, and instead ramp them up from baseline of X, with Y connections per second within next Z seconds (for example - 60 seconds). Of course, we could potentially recommend doing *both* of the tests, to show peak and nominal performance, so the lab test team would be able to asses results in wider spectrum of operating regimes.
>
>         --
>
>         ./
>
>             On 1 Jun 2021, at 05:35, Keiichi SHIMA<[e-mail]>  <mailto:[e-mail]>  wrote:
>
>             Edwin
>
>             Thank you for your comments.
>
>             We need to decide which part of the performance measurement we are trying to achieve. draft-lencse-bmwg-benchmarking-stateful focuses on the forwarding performance of NATxy boxes. As you have pointed our in your response, there are difference between stateless and statefull NATxy boxes, that is connection state management.
>
>             I agree your opinion that we need to consider the difference.
>
>             I also think TCP performance is an important metrics, however, it should/can be done in a different context. There are many approaches to measure TCP performance out there, so I think we can leave that part to the existing methods. I believe it is a good approach to focus on the functions unique to stateful NATxy boxes in this draft.
>
>             In the current draft, we have already introduced connection state management consideration (in the draft, we call it as the preliminary phase test). As you say, it is focusing on the UDP-oriented state management.
>
>             We need to discuss if this is enough for the case of TCP or not, and if it is not, then we need to invent a method to measure TCP-oritented state management performance of a NATxy box.
>
>             Regards,
>
>             ---
>
>             Keiichi SHIMA (島  慶一)
>
>             WIDE project<[e-mail]>  <mailto:[e-mail]>
>
>                PGP: 9D95 8544 A5CE D530 9230  DF1C BB6E ABE1 D91F EDFC
>
>             Research Laboratory, IIJ Innovation Institute, Inc<[e-mail]>  <mailto:[e-mail]>
>
>                PGP: 1B15 ACBC 1FB8 7325 B9BF  DCE6 B74D 1CFB 5173 3681
>
>                 On May 26, 2021, at 19:56, Edwin Cordeiro<[e-mail]>  <mailto:[e-mail]>  wrote:
>
>                 I understand that previous testing recommended using UDP instead of TCP and it was not my intent to say we need to replace UDP with TCP.
>
>                 But we have to consider if the goal of the new testing is just to have metrics to compare the transition techniques or do we want to also evaluate their scalability.
>
>                 As I believe we also want to test the scalability, specially on the comparison between stateless versus stateful alternatives, testing with TCP seems necessary. To test scalability, we need to consider scenarios that are closer to the real usage of such techniques. Therefore I'm not suggesting we scrap UDP tests, but that we should add some TCP based tests to cover such scenarios.
>
>                 When testing for scalability with TCP, we would target answering things like:
>
>                 - Does the transition technique affect how fast the congestion window grows?
>
>                 - Does it affect encryption protocols?
>
>                 - Is it impacted by out of order or lost packets?
>
>                 Best regards,
>
>                 Edwin Cordeiro
>
>                 On Tue, May 25, 2021 at 10:12 PM Gábor LENCSE<[e-mail]>  <mailto:[e-mail]>  wrote:
>
>                 Dear Edwin Cordeiro,
>
>                 Thank you very much for your review and support!
>
>                 I would like to note regarding testing with TCP that all RFC 2544, RFC 5180 and also RFC 8219 recommends testing with UDP.
>
>                 Additional testing with TCP seems to be feasible, but it requires some considerations and design decisions. For example, whereas you can simply send UDP datagrams as test traffic, but you need to establish a TCP connection first. The 3-way handshake may be done during the preliminary phase, and one can send TCP segments as test traffic in the real test phase.
>
>                 However, it is does not seem trivial to me, how deeply the Tester should follow the TCP protocol.
>
>                 On the one hand, perfectly implementing all the rules of the TCP protocol would impose a lot of limitations on the Tester, thus it would not be able to send test frames at the required rates. This would prohibit the execution of the benchmarking tests like throughput, frame loss rate, latency etc, :-(
>
>                 On the other hand, normal routers do not look into the TCP protocol header, but NAT boxes do. Thus sending TCP segments with improper Sequence number, Acknowledgement number, etc. fields could result in different problems in the DUT depending on how deeply it follows the TCP protocol.
>
>                 Best regards,
>
>                 Gábor
>
>                 24/05/2021 13:51 keltezéssel, Edwin Cordeiro írta:
>
>                     Dear authors of draft-lencse-bmwg-benchmarking-stateful,
>
>                     I agree with the motivation and ideas of the draft and I support its adoption.
>
>                     I recommend that the TCP considerations for such benchmarking methodology should not be left for a future document, as TCP still is the most used protocol. So, the details on TCP difficulties and an extension of this benchmark to consider TCP, should be in the scope of this document.
>
>                     Best regards,
>
>                     Edwin Cordeiro
>
>                     _______________________________________________
>
>                     bmwg mailing list
>
>                     [e-mail]  <mailto:[e-mail]>
>
>                     https://www.ietf.org/mailman/listinfo/bmwg  <https://www.ietf.org/mailman/listinfo/bmwg>
>
>                 _______________________________________________
>
>                 bmwg mailing list
>
>                 [e-mail]  <mailto:[e-mail]>
>
>                 https://www.ietf.org/mailman/listinfo/bmwg  <https://www.ietf.org/mailman/listinfo/bmwg>
>
>                 _______________________________________________
>
>                 bmwg mailing list
>
>                 [e-mail]  <mailto:[e-mail]>
>
>                 https://www.ietf.org/mailman/listinfo/bmwg  <https://www.ietf.org/mailman/listinfo/bmwg>
>
>             _______________________________________________
>
>             bmwg mailing list
>
>             [e-mail]  <mailto:[e-mail]>
>
>             https://www.ietf.org/mailman/listinfo/bmwg  <https://www.ietf.org/mailman/listinfo/bmwg>
>
>
>
>     _______________________________________________
>
>     bmwg mailing list
>
>     [e-mail]  <mailto:[e-mail]>
>
>     https://www.ietf.org/mailman/listinfo/bmwg  <https://www.ietf.org/mailman/listinfo/bmwg>
>
>
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/bmwg