Re: [bmwg] draft-lencse-bmwg-benchmarking-stateful

[Łukasz Bromirski <[e-mail]>]

Keiichi,

There are two things we need to consider here.

First of all, with recent QUIC RFC, world will move slowly but surely to UDP as part of HTTP/3. We won’t see this happening massively this year, but so won’t IPv6 adoption.

However, until that, TCP stays major protocol used by essentially everything, including - recently to bigger degree - DNS (DNSSEC, DoT & DoH).

The issue with any middle box is that vendors typically optimize processing for specific protocol. While measuring performance for UDP is great for gaming and voice (typically), TCP brings additional processing overhead which will make the box (typically) perform slower.

As for the concerns on testing - we can introduce concept of „ramp up” during test, which is to lessen the impact of blasting DUT with maximum amount of flows at the same time, and instead ramp them up from baseline of X, with Y connections per second within next Z seconds (for example - 60 seconds). Of course, we could potentially recommend doing *both* of the tests, to show peak and nominal performance, so the lab test team would be able to asses results in wider spectrum of operating regimes.

-- 
./

> On 1 Jun 2021, at 05:35, Keiichi SHIMA <[e-mail]> wrote:
> 
> Edwin
> 
> Thank you for your comments.
> 
> We need to decide which part of the performance measurement we are trying to achieve. draft-lencse-bmwg-benchmarking-stateful focuses on the forwarding performance of NATxy boxes. As you have pointed our in your response, there are difference between stateless and statefull NATxy boxes, that is connection state management.
> 
> I agree your opinion that we need to consider the difference.
> 
> I also think TCP performance is an important metrics, however, it should/can be done in a different context. There are many approaches to measure TCP performance out there, so I think we can leave that part to the existing methods. I believe it is a good approach to focus on the functions unique to stateful NATxy boxes in this draft.
> 
> In the current draft, we have already introduced connection state management consideration (in the draft, we call it as the preliminary phase test). As you say, it is focusing on the UDP-oriented state management.
> 
> We need to discuss if this is enough for the case of TCP or not, and if it is not, then we need to invent a method to measure TCP-oritented state management performance of a NATxy box.
> 
> Regards,
> 
> ---
> Keiichi SHIMA (島 慶一)
> WIDE project <[e-mail]>
>    PGP: 9D95 8544 A5CE D530 9230  DF1C BB6E ABE1 D91F EDFC
> Research Laboratory, IIJ Innovation Institute, Inc <[e-mail]>
>    PGP: 1B15 ACBC 1FB8 7325 B9BF  DCE6 B74D 1CFB 5173 3681
> 
> 
> 
> 
> 
> 
>> On May 26, 2021, at 19:56, Edwin Cordeiro <[e-mail]> wrote:
>> 
>> I understand that previous testing recommended using UDP instead of TCP and it was not my intent to say we need to replace UDP with TCP.
>> 
>> But we have to consider if the goal of the new testing is just to have metrics to compare the transition techniques or do we want to also evaluate their scalability.
>> 
>> As I believe we also want to test the scalability, specially on the comparison between stateless versus stateful alternatives, testing with TCP seems necessary. To test scalability, we need to consider scenarios that are closer to the real usage of such techniques. Therefore I'm not suggesting we scrap UDP tests, but that we should add some TCP based tests to cover such scenarios.
>> 
>> When testing for scalability with TCP, we would target answering things like:
>> - Does the transition technique affect how fast the congestion window grows?
>> - Does it affect encryption protocols?
>> - Is it impacted by out of order or lost packets?
>> 
>> Best regards,
>> 
>> Edwin Cordeiro
>> 
>> 
>> On Tue, May 25, 2021 at 10:12 PM Gábor LENCSE <[e-mail]> wrote:
>> Dear Edwin Cordeiro,
>> 
>> Thank you very much for your review and support!
>> 
>> I would like to note regarding testing with TCP that all RFC 2544, RFC 5180 and also RFC 8219 recommends testing with UDP.
>> 
>> Additional testing with TCP seems to be feasible, but it requires some considerations and design decisions. For example, whereas you can simply send UDP datagrams as test traffic, but you need to establish a TCP connection first. The 3-way handshake may be done during the preliminary phase, and one can send TCP segments as test traffic in the real test phase.
>> However, it is does not seem trivial to me, how deeply the Tester should follow the TCP protocol.
>> On the one hand, perfectly implementing all the rules of the TCP protocol would impose a lot of limitations on the Tester, thus it would not be able to send test frames at the required rates. This would prohibit the execution of the benchmarking tests like throughput, frame loss rate, latency etc, :-(
>> On the other hand, normal routers do not look into the TCP protocol header, but NAT boxes do. Thus sending TCP segments with improper Sequence number, Acknowledgement number, etc. fields could result in different problems in the DUT depending on how deeply it follows the TCP protocol.
>> 
>> Best regards,
>> 
>> Gábor
>> 
>> 24/05/2021 13:51 keltezéssel, Edwin Cordeiro írta:
>>> Dear authors of draft-lencse-bmwg-benchmarking-stateful,
>>> 
>>> I agree with the motivation and ideas of the draft and I support its adoption.
>>> 
>>> I recommend that the TCP considerations for such benchmarking methodology should not be left for a future document, as TCP still is the most used protocol. So, the details on TCP difficulties and an extension of this benchmark to consider TCP, should be in the scope of this document.
>>> 
>>> Best regards,
>>> 
>>> Edwin Cordeiro
>>> 
>>> 
>>> _______________________________________________
>>> bmwg mailing list
>>> 
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/bmwg
>> 
>> _______________________________________________
>> bmwg mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/bmwg
>> _______________________________________________
>> bmwg mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/bmwg
> 
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/bmwg