Re: [bmwg] I-D Action: draft-ietf-bmwg-benchmarking-stateful-04.txt

[Gábor LENCSE <[e-mail]>]

Hi Vratko,

Thank you for sharing your experience!

Please see my answers inline.

10/31/2023 6:38 PM keltezéssel, Vratko Polak -X (vrpolak - PANTHEON TECH 
SRO at Cisco) írta:
> Hi Gabor.
>
> When preparing (my part of) MLRsearch presentation,
> I wanted to refer to my earlier comments
> on how do we (fd.io CSIT project)
> use MLRsearch when testing stateful NAT(44).

I nearly had my own experience with FD.io VPP stateful NAT64. When I was 
in Japan from September 15 to December 15, 2022, I tested various 
stateful NAT64 implementations. I also wanted to include FD.io VPP. 
However, it crashed. I asked for help on the vpp-dev mailing list, and I 
received replies and tried to debug the case, but it was too complicated 
and I had to leave it out as my time was limited.

Finally, I could test three stateful NAT64 implementations:
- Jool
- tayga+iptables
- OpenBSD PF

The aim was to validate the methodology, this is why I wanted to use 
radically different stateful NAT64 implementations. I will present some 
of the results at IETF118. If you are interested, you can find all the 
results and all the details of the tests in our (open access) paper: 
https://doi.org/10.1016/j.comcom.2023.08.009

However, I did not give it up benchmarking FD.io VPP stateful NAT64. I 
definitely plan that my team will measure the stateful NAT64 and/or 
stateful NAT44 performance of VPP in the near future, hopefully in this 
year.

> But it seems such comments never went past the planning stage
> (at least I do not see any relevant e-mails from me in bmwg archive),
> so now is a good time to write them down.
>
> There are some differences between what we did when creating our tests
> and what do you propose in your draft,
> stemming mainly from the fact that out traffic generator is less capable,
> but on the other hand we have better (not totally black box)
> access to the DUT.
>
> Firstly, we can configure memory usage of our DUT.
> Instead of discovering "connection tracking table capacity",
> we just tweaked the memory usage for each scale (number of flows)
> until the DUT stopped crashing. :)
> We still keep testing at few different scales,
> as the performance depends on various memory caches.

Of course, I could also do anything at the DUT, as they were always 
Linux servers. I willfully refrained from it to keep the black box approach.

> Most importantly, we can ask our DUT to tell us
> how many sessions it is tracking,
> so we do not do any verification trials.

I admit that sometimes I also logged the number of sessions after the 
trials to check that everything was OK, but I used the results only for 
checking. :-)

> Secondly, just by simple reconfiguration (disable and re-enable NAT),
> our DUT is able to delete the content of the connection tracking table
> so fast it is not worth measuring.
That is great!

However, it is not always so. For example, in the case of iptables, the 
deletion of the states is much slower that their establishment.

> In principle, the test could override the time needed
> for a connection tracking table entry to time out
> (after seeing last activity), but we were not forced to do that yet.
>
> Now, to bad news. Our traffic generator does not really have
> convenient enough state table on the Responder side.
> I mean, it can track UDP and even TCP sessions properly,
> but once the trial ends, all is forgotten.
> So we can do bidirectional traffic,
> but we cannot do unidirectional traffic from the Responder to the Initiator.
>
> Also, our control over port numbers is limited.
> Source port number is always randomly selected by the traffic generator,
> and the destination port number has to be just one value
> (unless I want to add one TG configuration item per each value separately).

If you want, you can use siitperf. It is a free software under GPLv3 
license. The execution of a single test trial is implemented in C/C++ 
using DPDK. The binary program takes a high number of parameters from 
the configuration file and command line. And the binary search is 
implemented as a bash shell script. I think it is enough if you only 
change or replace the bash shell script.

If you would like to try it, I am glad to help you!  I would be happy if 
you could use siitperf for performing MLRsearch!


> Now, MLRsearch wants to control both the trial load and the trial duration.
> That may work for Phase 2 if the number of flows is small,
> but it will not work for Phase 1.
> There, number of frames is determined by the number of flows,
> so the trial duration is computed from that and the trial load.
> So I taught MLRsearch library how to deal with that.
I feel that siitperf could be a very handy tool for you. The user has 
full control over the source and destination port number ranges (that 
is, the number of network flows) and the frame rate can be set 
separately for phase 1 and phase 2.

> There are also other subtle differences between your draft and CSIT.
> For phase 1, we do not count the frames arriving at the Responder,
> we count "confirmed sessions" on the Initiator side.
> One reason is that a real client would not be happy to see zero frames.
> Also, our DUT is tracking sessions differently when they are only half-open,
> (at least for TCP) and we wanted a slightly more realistic scenario.
> Of course, we use the fast table drop before each subsequent trial.
>
> For phase 2 tests, we have separate suites,
> and we do not do full Connection Establishment Rate search there.
> Similarly to the memory situation,
> we just hardcoded the loads that proved to be safe and low before,
> and use the DUT session count to confirm that is still the case.
> In general, we use somewhat heavier transactions for Phase 2,
> but still way shorter than what realistic sessions would be.
> Also, we only use simple L2 counters to see
> the expected number of frames has been exchanged.
> This allows us to reach loads where the protocol-level counters
> become unreliable due to the traffic generator running out
> of CPU cycles to keep them properly updated.
>
> Also, we try to be smart with tracking sessions
> possibly timing out on DUT during the next trial.
> Firstly, every time we see a trial with zero loss,
> we assume each timer has been reset.
> If we suspect some sessions may time out during next trial,
> we execute one of those hardcoded trials with known safe value.
> (In principle. In practice we do not test scales that need that anymore.)
> After each trial we should confirm DUT still tracks all the sessions,
> but we do that only at the end of the search,
> as I do not remember that check ever failing.
>
> I think all the other tricks we apply
> are outside the scope of your draft
> (e.g. what to do when the traffic generator
> cannot sustain the required load).
I can handle it easily. Before starting a real test where the Tester 
benchmarks a DUT, I perform the _self-test of the Tester_. To that end, 
the DUT is omitted and the Tester is looped back. The tests (maximum 
connection establishment rate, throughput) are executed. The results 
show the capabilities of the tester. Then the tester can be used for 
testing the DUT up to 90% of the measured rates.

Will you come to IETF 118 in person?

If so, then perhaps it would be useful to sit down and discuss in person 
how you could use siitperf for MLRsearch. :-)

Best regards,

Gábor


> Vratko.
>
> -----Original Message-----
> From: bmwg<[e-mail]>  On Behalf Of Gábor LENCSE
> Sent: Wednesday, 13 September, 2023 08:31
> To:[e-mail]
> Subject: Re: [bmwg] I-D Action: draft-ietf-bmwg-benchmarking-stateful-04.txt
>
> Dear BMWG Chairs and Members,
>
> I have just posted the updated version of our I-D.  There were only some
> minor updates to Section 3.2 and Section 7.
>
> With my co-author, Keiichi Shima, we believe that our draft is ready for
> WGLC.
>
> Best regards,
>
> Gábor
>
>
> 9/13/2023 8:26 AM keltezéssel,[e-mail]  írta:
>> Internet-Draft draft-ietf-bmwg-benchmarking-stateful-04.txt is now available.
>> It is a work item of the Benchmarking Methodology (BMWG) WG of the IETF.
>>
>>      Title:   Benchmarking Methodology for Stateful NATxy Gateways using RFC 4814 Pseudorandom Port Numbers
>>      Authors: Gábor Lencse
>>               Keiichi Shima
>>      Name:    draft-ietf-bmwg-benchmarking-stateful-04.txt
>>      Pages:   28
>>      Dates:   2023-09-12
>>
>> Abstract:
>>
>>      RFC 2544 has defined a benchmarking methodology for network
>>      interconnect devices.  RFC 5180 addressed IPv6 specificities and it
>>      also provided a technology update, but excluded IPv6 transition
>>      technologies.  RFC 8219 addressed IPv6 transition technologies,
>>      including stateful NAT64.  However, none of them discussed how to
>>      apply RFC 4814 pseudorandom port numbers to any stateful NATxy
>>      (NAT44, NAT64, NAT66) technologies.  We discuss why using
>>      pseudorandom port numbers with stateful NATxy gateways is a difficult
>>      problem.  We recommend a solution limiting the port number ranges and
>>      using two test phases (phase 1 and phase 2).  We show how the classic
>>      performance measurement procedures (e.g. throughput, frame loss rate,
>>      latency, etc.) can be carried out.  We also define new performance
>>      metrics and measurement procedures for maximum connection
>>      establishment rate, connection tear down rate and connection tracking
>>      table capacity measurements.
>>
>> The IETF datatracker status page for this Internet-Draft is:
>> https://datatracker.ietf.org/doc/draft-ietf-bmwg-benchmarking-stateful/
>>
>> There is also an HTMLized version available at:
>> https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-benchmarking-stateful-04
>>
>> A diff from the previous version is available at:
>> https://author-tools.ietf.org/iddiff?url2=draft-ietf-bmwg-benchmarking-stateful-04
>>
>> Internet-Drafts are also available by rsync at:
>> rsync.ietf.org::internet-drafts
>>
>>
>> _______________________________________________
>> bmwg mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/bmwg
>>
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/bmwg
>