[bmwg] RFC 4814 pseudorandom port numbers versus RoCEv2

Gábor LENCSE <lencse@hit.bme.hu> Tue, 29 June 2021 20:52 UTC

To: "bmwg@ietf.org" <bmwg@ietf.org>
From: Gábor LENCSE <lencse@hit.bme.hu>
Message-ID: <1001b7d2-f1c9-05c4-bfdb-092481e49bb5@hit.bme.hu>
Date: Tue, 29 Jun 2021 22:51:57 +0200
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="------------66FE5D92B41776D75C872602"
Content-Language: en-US
Received-SPF: pass (frogstar.hit.bme.hu: authenticated connection) receiver=frogstar.hit.bme.hu; client-ip=79.121.43.161; helo=[192.168.1.148]; envelope-from=lencse@hit.bme.hu; x-software=spfmilter 2.001 http://www.acme.com/software/spfmilter/ with libspf2-1.2.10;
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/3RfMYcr80DbXRbF_t5wggU7akGQ>
Subject: [bmwg] RFC 4814 pseudorandom port numbers versus RoCEv2
Precedence: list

Dear BMWG Members,

I wonder if you have already met the following problem.

RFC 4814 requires the usage of pseudorandom source and destination port 
numbers. The recommended range for the destination port numbers is: [1, 
49151].
However the 4791 destination port number identifies RoCEv2 ( 
https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet ).

If you use a NIC that supports RoCEv2 (and it is enabled) then about 20 
test frames disappear from every 1 million test frames. Here are the 
results of a 60s long throughput test at 4,000,000fps rate:

root@hp1:~/siitperf# ./build/siitperf-tp 84 4000000 60 2000 2 2
EAL: Detected 32 lcore(s)
EAL: No free hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: PCI device 0000:5d:00.0 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
PMD: Broadcom Cumulus driver bnxt
PMD: 1.10.1:214.4.91
PMD: Driver HWRM version: 1.5.1
PMD: bnxt found at mem e6e10000, node addr 0x7f56c0000000M
EAL: PCI device 0000:5d:00.1 on NUMA socket 0
EAL:   probe driver: 14e4:16d7 net_bnxt
PMD: 1.10.1:214.4.91
PMD: Driver HWRM version: 1.5.1
PMD: bnxt found at mem e6e00000, node addr 0x7f56c0112000M
PMD: Port 0 Link Down
PMD: Port 0 Link Up - speed 10000 Mbps - full-duplex
PMD: Port 1 Link Down
PMD: Port 1 Link Up - speed 10000 Mbps - full-duplex
Info: Left port and Left Sender CPU core belong to the same NUMA node: 0
Info: Right port and Right Receiver CPU core belong to the same NUMA node: 0
Info: Right port and Right Sender CPU core belong to the same NUMA node: 0
Info: Left port and Left Receiver CPU core belong to the same NUMA node: 0
Info: Testing started.
Info: Reverse sender's sending took 59.9999998300 seconds.
Reverse frames sent: *240000000*
Info: Forward sender's sending took 59.9999999983 seconds.
Forward frames sent: 240000000
Reverse frames received: *239995097*
Forward frames received: 239995159
Info: Test finished.
root@hp1:~/siitperf# echo $(((240000000-239995097)/240))
*20*

That is, about 20 frames are lost from every 1,000,000 frames. (First, I 
have found it very strange. Using fixed port numbers, there was no frame 
loss, but using pseudorandom port numbers, 20 of 1,000,000 frames were 
lost even at significantly lower frame rates.)

I have debugged the issue by halving the destination port range, and I 
have found that if the destination port number range was set to [4791, 
4791] then ALL test frames were lost!

I just wanted to save you a few hours of debugging by letting you know 
this issue. :-)

Best regards,

Gábor

[bmwg] RFC 4814 pseudorandom port numbers versus … Gábor LENCSE