Re: [bmwg] Question regarding the updating of RFC 2544 with the usage of pseudorandom IP addresses

[Vasilenko Eduard <[e-mail]>]

Hi Gabor,
Many specialized network devices (especially middleboxes) are based on RISC processors.
They have many more cores than 32, it is already close to thousands (at least a few hundred are typical).
Hence, “not enough flows” may affect them very much.

But why only this problem is considered for RFC2544 refreshment?
I could propose a couple of additional:
- modern PFEs are capable of distributing resources between ports/interfaces in a very flexible way, especially for headers processing.
It does not make sense to test one port, not at all. At least one PFE (the latest is 51.2Tbps) should be fully loaded to see meaningful results.
- RFC6201 has a discussion that reset could be partial, software, hardware
- RFC9004 has a much deeper discussion on buffer’s size detection.
- and so on

IMHO: if refresh RFC2544 then refresh it all.
But what would happen with this RFC if half of BMWG documents were inserted into it?
Hence, the balance is needed for how much to insert and how much to reference.

Specifically for pseudorandom IP addresses:
It is possible to squeeze the text into 1 page and add it to the primary RFC.

The same is true for RFC 6201: it is possible to move all essence into RFC 2544bis.

But it would be very difficult to insert the essence of RFC 9004 into RFC 2544bis – too big text. Hence, the reference would be better.

Eduard
From: bmwg [mailto:[e-mail]] On Behalf Of Gabor LENCSE
Sent: Sunday, October 1, 2023 6:54 PM
To: [e-mail]
Subject: [bmwg] Question regarding the updating of RFC 2544 with the usage of pseudorandom IP addresses


Dear BMWG Members,

Previously I have consulted with the BMWG chairs about the following issue and they recommended that I should bring this question to the mailing list.

I have encountered with the following issue regarding the benchmarking of the packet forwarding performance of routers.

1. RFC 2544 defined a test frame format using fixed IP addresses and port numbers: https://datatracker.ietf.org/doc/html/rfc2544#appendix-C.2.6.4 (Some old network performance testers literally follow the above test frame format and they send always the very same test frames.)

2. As time passed by, it became common that routers have multiple processing units. (E.g., when a router is implemented by a Linux server, it is common that is has 32 or even more CPU cores.) Internet traffic contains multiple IP addresses and port numbers and RSS (Receive-Side Scaling) uses their entropy to distribute the traffic among the processing units.

3. To keep up with the development of the network interconnect devices, RFC 4814 recommended the usage of pseudorandom port numbers (using uniform distribution over well defined wide ranges): https://datatracker.ietf.org/doc/html/rfc4814#section-4.5

Unless multiple destination networks are used, it means that Testers use a single IP address pair (e.g. 198.18.0.2 and 198.19.0.2) and a high number of pseudorandom source and destination port numbers from the ranges recommended by RFC 4814.

In practice, this solution sometimes works well, sometimes not. For example, Under Debian Linux (at least from version 9 to version 11), the default setting of the RSS is that only the source IP address and destination IP address are included in the tuple for hashing.

root@dut:~# ethtool -n eno1 rx-flow-hash udp4
UDP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA

With these settings, the usage of pseudorandom port numbers is absolutely useless. However, one can set RSS so that the port number are also included in the tuple for hashing:

root@dut:~# ethtool -N eno1 rx-flow-hash udp4 sdfn
root@dut:~# ethtool -n eno1 rx-flow-hash udp4
UDP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

With the latter settings, the usage of a single IP address pair and pseudorandom port numbers works perfectly.

Under OpenBSD, the default behavior is the same (RSS considers only IP addresses), but there is no such tool that could be used to change it to use also the port numbers: https://marc.info/?l=openbsd-misc&m=166581934723445&w=2

It means that pseudorandom port numbers do not help. This RSS implementation would require the usage of pseudorandom IP addresses.

My suggestion is that we should update RFC 2544 (and RFC 5180 regarding IPv6) to support the usage of multiple, pseudorandom IP addresses in a similar manner as it was recommended regarding the port numbers by RFC 4814.

I have specific suggestions, and also working code: I have implemented the usage of pseudorandom IP addresses in the latest version of siitperf: https://github.com/lencsegabor/siitperf

I am planning to submit and Internet Draft about my recommendation, but before that I would like to know your opinion.

Do you find this idea meaningful?

It is also an important question, how the different router implementations behave. If there are several RSS implementations that use only the IP addresses (and not the port numbers) for hashing, then, IMHO, it is worth working on this update. However, if OpenBSD is the only such exception, then perhaps not.

So, please write me back if you have some knowledge about how commercial routers implement RSS.

Thank you very much for your feedback in advance!

Best regards,

Gábor