[bmwg] Question regarding the updating of RFC 2544 with the usage of pseudorandom IP addresses

[Gabor LENCSE <[e-mail]>]

Dear BMWG Members,

Previously I have consulted with the BMWG chairs about the following 
issue and they recommended that I should bring this question to the 
mailing list.

I have encountered with the following issue regarding the benchmarking 
of the packet forwarding performance of routers.

1. RFC 2544 defined a test frame format using fixed IP addresses and 
port numbers: 
https://datatracker.ietf.org/doc/html/rfc2544#appendix-C.2.6.4 (Some old 
network performance testers literally follow the above test frame format 
and they send always the very same test frames.)

2. As time passed by, it became common that routers have multiple 
processing units. (E.g., when a router is implemented by a Linux server, 
it is common that is has 32 or even more CPU cores.) Internet traffic 
contains multiple IP addresses and port numbers and RSS (Receive-Side 
Scaling) uses their entropy to distribute the traffic among the 
processing units.

3. To keep up with the development of the network interconnect devices, 
RFC 4814 recommended the usage of pseudorandom port numbers (using 
uniform distribution over well defined wide ranges): 
https://datatracker.ietf.org/doc/html/rfc4814#section-4.5

Unless multiple destination networks are used, it means that Testers use 
a single IP address pair (e.g. 198.18.0.2 and 198.19.0.2) and a high 
number of pseudorandom source and destination port numbers from the 
ranges recommended by RFC 4814.

In practice, this solution sometimes works well, sometimes not. For 
example, Under Debian Linux (at least from version 9 to version 11), the 
default setting of the RSS is that only the source IP address and 
destination IP address are included in the tuple for hashing.

root@dut:~# ethtool -n eno1 rx-flow-hash udp4
UDP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA

With these settings, the usage of pseudorandom port numbers is 
absolutely useless. However, one can set RSS so that the port number are 
also included in the tuple for hashing:

root@dut:~# ethtool -N eno1 rx-flow-hash udp4 sdfn
root@dut:~# ethtool -n eno1 rx-flow-hash udp4
UDP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

With the latter settings, the usage of _a single IP address pair and 
pseudorandom port numbers_ works perfectly.

Under OpenBSD, the default behavior is the same (RSS considers only IP 
addresses), but there is no such tool that could be used to change it to 
use also the port numbers: 
https://marc.info/?l=openbsd-misc&m=166581934723445&w=2 
<https://marc.info/?l=openbsd-misc&m=166581934723445&w=2>

It means that pseudorandom port numbers do not help. This RSS 
implementation would require the usage of _pseudorandom IP addresses_.

My suggestion is that we should update RFC 2544 (and RFC 5180 regarding 
IPv6) to support the usage of multiple, pseudorandom IP addresses in a 
similar manner as it was recommended regarding the port numbers by RFC 4814.

I have specific suggestions, and also working code: I have implemented 
the usage of pseudorandom IP addresses in the latest version of 
siitperf: https://github.com/lencsegabor/siitperf

I am planning to submit and Internet Draft about my recommendation, but 
before that I would like to know your opinion.

Do you find this idea meaningful?

It is also an important question, how the different router 
implementations behave. If there are several RSS implementations that 
use only the IP addresses (and not the port numbers) for hashing, then, 
IMHO, it is worth working on this update. However, if OpenBSD is the 
only such exception, then perhaps not.

So, please write me back if you have some knowledge about how commercial 
routers implement RSS.

Thank you very much for your feedback in advance!

Best regards,

Gábor