Re: [L4s-discuss] rpi 5 and hardware ptp

[rjmcmahon <[e-mail]>]

On 2024-04-08 12:32, rjmcmahon wrote:
>> 
>> Your kernel should have full support for ECN and PACING.
>> sysctl net.ipv4.tcp_ecn
>> to check for ecn state, 2 is the default, you probably want to
>> flip this to 1 on the end running iperf -s to get a ecn connection.
>> 
>> PACING is a setsockopt, try man tc or man fq and search for pacing.
>> 
For completeness, I added inflight bytes & packets (InF column) and cwnd 
packets to the pacing outputs. I also applied the following kernel 
settings. Then did a run w/TCP_NOTSENTLOWAT and without it.

Per the L4S designers:

The TCP memory buffers need to be big enough to avoid application and 
receive window limits (replace or set $SRC_IF to the NIC names on all 
nodes that carry the load traffic):

sudo sysctl -w net.core.wmem_default=$((100*1024*1024))
sudo sysctl -w net.core.wmem_max=$((100*1024*1024))
sudo sysctl -w net.core.rmem_default=$((100*1024*1024))
sudo sysctl -w net.core.rmem_max=$((100*1024*1024))
sudo sysctl -w net.ipv4.tcp_wmem="$((6*1024*1024)) $((100*1024*1024)) 
$((100*1024*1024))"
sudo sysctl -w net.ipv4.tcp_rmem="$((6*1024*1024)) $((100*1024*1024)) 
$((100*1024*1024))"

For L4S Prague traffic turn off the TSO/GSO/GRO/LRO to avoid large 
bursts
sudo ethtool -K $SRC_IF tso off gso off gro off lro off

On the clients and servers that use TCP-Prague, use the “fq” qdisc 
(instead of the “fq_codel” default) to ensure pacing can be done 
correctly
sudo tc qdisc replace dev $SRC_IF root handle 1: fq limit 100000 
flow_limit 50000

The latest iperf 2 output

rjmcmahon@fedora:~/Code/master/iperf2-code$ src/iperf -c 192.168.1.34 -i 
1 --fq-rate-step 100m --fq-rate-step-interval 1 --trip-times -t 15 -w 4m
------------------------------------------------------------
Client connecting to 192.168.1.34, TCP port 5001 with pid 74072 (1/0 
flows/load)
Write buffer size: 131072 Byte
fair-queue socket pacing set to  100 Mbit/s (stepping rate by  100 
Mbit/s)
TCP congestion control using cubic
TOS defaults to 0x0 (dscp=0,ecn=0) (Nagle on)
TCP window size: 7.63 MByte (WARNING: requested 3.81 MByte)
Event based writes (pending queue watermark at 16384 bytes)
------------------------------------------------------------
[  1] local 192.168.1.103%enp4s0 port 48874 connected with 192.168.1.34 
port 5001 (prefetch=16384) (trip-times) (sock=3) 
(icwnd/mss/irtt=14/1448/187) (ct=0.24 ms) on 2024-04-08 12:50:51.354 
(PDT)
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     
InF(pkts)/Cwnd(pkts)/RTT(var)  fq-rate  NetPwr
[  1] 0.00-1.00 sec  12.0 MBytes   101 Mbits/sec  96/0        23       
39K(28)/41K(29)/244(44) us  100 Mbit/sec 51570
[  1] 1.00-2.00 sec  23.6 MBytes   198 Mbits/sec  189/0        63       
38K(27)/38K(27)/178(18) us  200 Mbit/sec 139172
[  1] 2.00-3.00 sec  35.5 MBytes   298 Mbits/sec  284/0        76       
39K(28)/39K(28)/183(33) us  300 Mbit/sec 203412
[  1] 3.00-4.00 sec  47.5 MBytes   398 Mbits/sec  380/0       104       
53K(38)/53K(38)/187(37) us  400 Mbit/sec 266350
[  1] 4.00-5.00 sec  59.2 MBytes   497 Mbits/sec  474/0       180       
49K(35)/53K(38)/195(31) us  500 Mbit/sec 318606
[  1] 5.00-6.00 sec  71.2 MBytes   598 Mbits/sec  570/0       221       
39K(28)/39K(28)/267(151) us  600 Mbit/sec 279817
[  1] 6.00-7.00 sec  83.0 MBytes   696 Mbits/sec  664/0       275       
43K(31)/45K(32)/208(42) us  700 Mbit/sec 418422
[  1] 7.00-8.00 sec  94.9 MBytes   796 Mbits/sec  759/0       336       
33K(24)/33K(24)/198(38) us  800 Mbit/sec 502443
[  1] 8.00-9.00 sec   107 MBytes   894 Mbits/sec  853/0       397       
32K(23)/38K(27)/211(36) us  900 Mbit/sec 529879
[  1] 9.00-10.00 sec   111 MBytes   930 Mbits/sec  887/0       340       
21K(15)/21K(15)/310(190) us 1.00 Gbit/sec 375035
[  1] 10.00-11.00 sec   111 MBytes   931 Mbits/sec  888/0       523      
  28K(20)/29K(21)/242(28) us 1.10 Gbit/sec 480958
[  1] 11.00-12.00 sec   111 MBytes   932 Mbits/sec  889/0       617      
  21K(15)/26K(19)/204(21) us 1.20 Gbit/sec 571191
[  1] 12.00-13.00 sec   111 MBytes   929 Mbits/sec  886/0       592      
  36K(26)/36K(26)/360(230) us 1.30 Gbit/sec 322583
[  1] 13.00-14.00 sec   111 MBytes   930 Mbits/sec  887/0       605      
  28K(20)/29K(21)/254(31) us 1.40 Gbit/sec 457720
[  1] 14.00-15.00 sec   111 MBytes   933 Mbits/sec  890/0       610      
  22K(16)/24K(17)/211(13) us 1.50 Gbit/sec 552863
[  1] 0.00-15.01 sec  1.17 GBytes   670 Mbits/sec  9597/0      4962      
  24K/210(11) us 399026

rjmcmahon@fedora:~/Code/master/iperf2-code$ src/iperf -c 192.168.1.34 -i 
1 --fq-rate-step 100m --fq-rate-step-interval 1 --trip-times -t 15 -w 4m 
--tcp-write-prefetch 0
------------------------------------------------------------
Client connecting to 192.168.1.34, TCP port 5001 with pid 74082 (1/0 
flows/load)
Write buffer size: 131072 Byte
fair-queue socket pacing set to  100 Mbit/s (stepping rate by  100 
Mbit/s)
TCP congestion control using cubic
TOS defaults to 0x0 (dscp=0,ecn=0) (Nagle on)
TCP window size: 7.63 MByte (WARNING: requested 3.81 MByte)
------------------------------------------------------------
[  1] local 192.168.1.103%enp4s0 port 54034 connected with 192.168.1.34 
port 5001 (trip-times) (sock=3) (icwnd/mss/irtt=14/1448/288) (ct=0.35 
ms) on 2024-04-08 12:51:26.722 (PDT)
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     
InF(pkts)/Cwnd(pkts)/RTT(var)  fq-rate  NetPwr
[  1] 0.00-1.00 sec  17.6 MBytes   148 Mbits/sec  141/0        35       
25K(18)/33K(24)/349(264) us  100 Mbit/sec 52955
[  1] 1.00-2.00 sec  23.0 MBytes   193 Mbits/sec  184/0         0       
25K(18)/50K(36)/201(31) us  200 Mbit/sec 119986
[  1] 2.00-3.00 sec  35.6 MBytes   299 Mbits/sec  285/0        81       
42K(30)/42K(30)/195(26) us  300 Mbit/sec 191567
[  1] 3.00-4.00 sec  48.4 MBytes   406 Mbits/sec  387/0       123       
33K(24)/33K(24)/163(8) us  400 Mbit/sec 311195
[  1] 4.00-5.00 sec  58.9 MBytes   494 Mbits/sec  471/0       173       
36K(26)/36K(26)/195(60) us  500 Mbit/sec 316589
[  1] 5.00-6.00 sec  69.2 MBytes   581 Mbits/sec  554/0       185       
38K(27)/38K(27)/351(302) us  600 Mbit/sec 206877
[  1] 6.00-7.00 sec  83.4 MBytes   699 Mbits/sec  667/0       278       
32K(23)/32K(23)/216(52) us  700 Mbit/sec 404745
[  1] 7.00-8.00 sec  94.6 MBytes   794 Mbits/sec  757/0       347       
52K(37)/52K(37)/226(42) us  800 Mbit/sec 439033
[  1] 8.00-9.00 sec   106 MBytes   891 Mbits/sec  850/0       376       
33K(24)/35K(25)/214(33) us  900 Mbit/sec 520613
[  1] 9.00-10.00 sec   112 MBytes   941 Mbits/sec  897/0       367       
18K(13)/26K(19)/194(31) us 1.00 Gbit/sec 606039
[  1] 10.00-11.00 sec   110 MBytes   921 Mbits/sec  878/0       515      
  18K(13)/26K(19)/207(21) us 1.10 Gbit/sec 555948
[  1] 11.00-12.00 sec   110 MBytes   920 Mbits/sec  877/0       596      
  19K(14)/26K(19)/223(26) us 1.20 Gbit/sec 515471
[  1] 12.00-13.00 sec   112 MBytes   942 Mbits/sec  898/0       643      
  16K(12)/24K(17)/213(23) us 1.30 Gbit/sec 552595
[  1] 13.00-14.00 sec   110 MBytes   920 Mbits/sec  877/0       646      
  18K(13)/24K(17)/197(73) us 1.40 Gbit/sec 583503
[  1] 14.00-15.00 sec   112 MBytes   941 Mbits/sec  897/0       607      
  24K(17)/26K(19)/211(27) us 1.50 Gbit/sec 557211
[  1] 0.00-15.08 sec  1.17 GBytes   669 Mbits/sec  9621/0      5000      
  24K/205(15) us 407992

Bob