[L4s-discuss] Difference between InF and InP w/iperf 2

rjmcmahon <rjmcmahon@rjmcmahon.com> Mon, 25 March 2024 23:58 UTC

Return-Path: <rjmcmahon@rjmcmahon.com>
X-Original-To: l4s-discuss@ietfa.amsl.com
Delivered-To: l4s-discuss@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CF125C18DBA4 for <l4s-discuss@ietfa.amsl.com>; Mon, 25 Mar 2024 16:58:01 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.105
X-Spam-Level:
X-Spam-Status: No, score=-2.105 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HK_RANDOM_ENVFROM=0.001, HK_RANDOM_FROM=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=rjmcmahon.com
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KzHIGYboc-0E for <l4s-discuss@ietfa.amsl.com>; Mon, 25 Mar 2024 16:57:57 -0700 (PDT)
Received: from bobcat.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C4267C151066 for <l4s-discuss@ietf.org>; Mon, 25 Mar 2024 16:57:57 -0700 (PDT)
Received: from mail.rjmcmahon.com (bobcat.rjmcmahon.com [45.33.58.123]) by bobcat.rjmcmahon.com (Postfix) with ESMTPA id 274EF1B258 for <l4s-discuss@ietf.org>; Mon, 25 Mar 2024 16:57:57 -0700 (PDT)
DKIM-Filter: OpenDKIM Filter v2.11.0 bobcat.rjmcmahon.com 274EF1B258
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rjmcmahon.com; s=bobcat; t=1711411077; bh=+1w45eQ9oZxShYqK+Lg9njQ34a+lo6EuKLP/rfEISck=; h=Date:From:To:Subject:From; b=n53ckoJcmIMngOUITSJlsQF2ZqlokX7iJu3RE8RfEgeSqnkIw0Zavfxr0Pix+sQzA ne8FU1S/ttwxvSkrqXALR8obyCNbQq8cP/hg+25eheWmJ3l17FVzaFlRPQ6SMzOeBt OEmZMaj68LXo9xv3lsifNRBL9GmUSWV+6XkCvZCI=
MIME-Version: 1.0
Date: Mon, 25 Mar 2024 16:57:57 -0700
From: rjmcmahon <rjmcmahon@rjmcmahon.com>
To: l4s-discuss@ietf.org
Message-ID: <216133302c2361838268517fe2cbe3b0@rjmcmahon.com>
X-Sender: rjmcmahon@rjmcmahon.com
Content-Type: text/plain; charset="US-ASCII"; format="flowed"
Content-Transfer-Encoding: 7bit
Archived-At: <https://mailarchive.ietf.org/arch/msg/l4s-discuss/kfCnglw8XhxWCMnHVZApK_yDc1o>
Subject: [L4s-discuss] Difference between InF and InP w/iperf 2
X-BeenThere: l4s-discuss@ietf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: "Low Latency, Low Loss, Scalable Throughput \(L4S\) " <l4s-discuss.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/l4s-discuss>, <mailto:l4s-discuss-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/l4s-discuss/>
List-Post: <mailto:l4s-discuss@ietf.org>
List-Help: <mailto:l4s-discuss-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/l4s-discuss>, <mailto:l4s-discuss-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Mar 2024 23:58:01 -0000

Hi All,

There is a subtle but important distinction between iperf 2's in 
progress stats. One is called InF for in-flight and computed by taking 
the stats from the stack. The second is called InP for in-progress and 
taken by computing the queue depth per Little's Law. The latter is from 
an app view. The following two runs illustrate the differences.

First a run where there is a very large send side buffer and 
TCP_NOTSENT_LOWAT is not used. In this case the send side socket buffer 
should build up, which it does, and the e2e delay is seen per InP but 
not InF. That's because InF is from the stack's view of the network 
while InP is from the application's view. Notice the InP is near the 
100Mbytes socket buffer size while the InF is about 1.5Mbytes

The second run sets TCP_NOTSENT_LOWAT and no the InF and InP are aligned 
at about 1.5Mbytes. The app gets much better performance with this 
socket option.

Run 1: No TCP_NOTSENT_LOWAT

rjmcmahon@fedora:~/Code/inflight/iperf2-code$ iperf -c 192.168.1.35 
--trip-times -e -i 1 --sync-transfer-id -w 50m --tcp-write-prefetch 0
------------------------------------------------------------
Client connecting to 192.168.1.35, TCP port 5001 with pid 240696 (1/0 
flows/load)
Write buffer size: 131072 Byte
TCP congestion control using cubic
TOS set to 0x0 (dscp=0,ecn=0) (Nagle on)
TCP window size: 95.4 MByte (WARNING: requested 47.7 MByte)
------------------------------------------------------------
[  1] local 192.168.1.103%enp4s0 port 44754 connected with 192.168.1.35 
port 5001 (trip-times) (sock=3) (icwnd/mss/irtt=14/1448/176) (ct=0.24 
ms) on 2024-03-25 16:50:54.655 (PDT)
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     
InF(pkts)/Cwnd/RTT(var)        NetPwr
[  1] 0.00-1.00 sec   192 MBytes  1.61 Gbits/sec  1533/0        61     
1538K(1088)/1555K/13234(110) us  15183
[  1] 1.00-2.00 sec  96.4 MBytes   808 Mbits/sec  771/0         0     
1667K(1179)/1674K/14273(81) us  7080
[  1] 2.00-3.00 sec   128 MBytes  1.08 Gbits/sec  1026/0         0     
1790K(1266)/1801K/15451(109) us  8704
[  1] 3.00-4.00 sec  96.2 MBytes   807 Mbits/sec  770/0         0     
1859K(1315)/1875K/16008(84) us  6305
[  1] 4.00-5.00 sec   128 MBytes  1.08 Gbits/sec  1026/0         2     
1340K(948)/1365K/11658(133) us  11535
[  1] 5.00-6.00 sec  96.2 MBytes   807 Mbits/sec  770/0         0     
1409K(997)/1448K/12301(98) us  8205
[  1] 6.00-7.00 sec   128 MBytes  1.08 Gbits/sec  1026/0         0     
1474K(1043)/1530K/13059(123) us  10298
[  1] 7.00-8.00 sec  96.2 MBytes   807 Mbits/sec  770/0         0     
1534K(1085)/1575K/13494(111) us  7479
[  1] 8.00-9.00 sec   128 MBytes  1.08 Gbits/sec  1027/0         0     
1602K(1133)/1614K/13807(132) us  9749
[  1] 9.00-10.00 sec  96.2 MBytes   807 Mbits/sec  770/0         0     
1593K(1127)/1636K/13980(130) us  7219
[  1] 10.00-10.57 sec   128 KBytes  1.83 Mbits/sec  1/0         0        
0K(0)/1665K/14358(111) us  15.90
[  1] 0.00-10.57 sec  1.16 GBytes   941 Mbits/sec  9490/0        63      
   0K(0)/1665K/14358(111) us  8193

root@rpi5-35:~# iperf -s -i 1 -e
------------------------------------------------------------
Server listening on TCP port 5001 with pid 35250
Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
TCP congestion control default cubic
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.35%eth0 port 5001 connected with 192.168.1.103 
port 44754 (trip-times) (sock=4) (peer 2.2.0-rc) 
(icwnd/mss/irtt=14/1448/182) on 2024-03-25 16:50:54.655 (PDT)
[ ID] Interval        Transfer    Bandwidth    Burst Latency 
avg/min/max/stdev (cnt/size) inP NetPwr  Reads=Dist
[  1] 0.00-1.00 sec   112 MBytes   941 Mbits/sec  
451.474/1.878/839.327/236.465 ms (897/131093)  171 MByte 260  
22019=21972:11:11:8:0:2:2:13
[  1] 1.00-2.00 sec   112 MBytes   941 Mbits/sec  
709.032/560.182/839.576/80.886 ms (898/131052) 69.9 MByte 166  
22841=22836:3:2:0:0:0:0:0
[  1] 2.00-3.00 sec   112 MBytes   942 Mbits/sec  
690.533/559.994/839.511/79.692 ms (897/131201) 90.0 MByte 170  
18997=18993:4:0:0:0:0:0:0
[  1] 3.00-4.00 sec   112 MBytes   942 Mbits/sec  
708.975/559.939/839.014/80.667 ms (898/131063) 69.8 MByte 166  
19325=19325:0:0:0:0:0:0:0
[  1] 4.00-5.00 sec   112 MBytes   941 Mbits/sec  
689.814/559.899/838.858/79.718 ms (898/131053) 90.0 MByte 171  
19552=19552:0:0:0:0:0:0:0
[  1] 5.00-6.00 sec   112 MBytes   941 Mbits/sec  
709.082/559.756/838.974/80.714 ms (898/131053) 69.8 MByte 166  
19050=19030:4:0:0:1:0:1:14
[  1] 6.00-7.00 sec   112 MBytes   941 Mbits/sec  
690.027/559.817/839.295/79.776 ms (898/131050) 90.0 MByte 171  
19573=19570:2:1:0:0:0:0:0
[  1] 7.00-8.00 sec   112 MBytes   942 Mbits/sec  
709.150/560.197/839.131/80.593 ms (898/131056) 69.8 MByte 166  
20616=20608:4:3:1:0:0:0:0
[  1] 8.00-9.00 sec   112 MBytes   941 Mbits/sec  
690.031/560.092/839.122/79.848 ms (898/131050) 90.0 MByte 171  
18866=18864:2:0:0:0:0:0:0
[  1] 9.00-10.00 sec   112 MBytes   942 Mbits/sec  
709.217/560.065/838.999/80.484 ms (898/131056) 69.8 MByte 166  
18885=18882:2:1:0:0:0:0:0
[  1] 10.00-10.57 sec  64.0 MBytes   941 Mbits/sec  
699.737/560.046/838.851/80.583 ms (512/131051) 69.8 MByte 168  
210924=11199:1:0:0:0:0:0:0
[  1] 0.00-10.57 sec  1.16 GBytes   941 Mbits/sec  
677.050/1.878/839.576/128.445 ms (9490/131072) 75.7 MByte 174  
210924=210831:33:18:9:1:2:3:27

Run 2: With TCP_NOTSENT_LOWAT

rjmcmahon@fedora:~/Code/inflight/iperf2-code$ iperf -c 192.168.1.35 
--trip-times -e -i 1 --sync-transfer-id -w 50m --tcp-write-prefetch 128K
------------------------------------------------------------
Client connecting to 192.168.1.35, TCP port 5001 with pid 240730 (1/0 
flows/load)
Write buffer size: 131072 Byte
TCP congestion control using cubic
TOS set to 0x0 (dscp=0,ecn=0) (Nagle on)
TCP window size: 95.4 MByte (WARNING: requested 47.7 MByte)
Event based writes (pending queue watermark at 131072 bytes)
------------------------------------------------------------
[  1] local 192.168.1.103%enp4s0 port 53192 connected with 192.168.1.35 
port 5001 (prefetch=131072) (trip-times) (sock=3) 
(icwnd/mss/irtt=14/1448/169) (ct=0.23 ms) on 2024-03-25 16:54:02.796 
(PDT)
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     
InF(pkts)/Cwnd/RTT(var)        NetPwr
[  1] 0.00-1.00 sec   114 MBytes   955 Mbits/sec  911/0        72     
1534K(1085)/1534K/13047(100) us  9152
[  1] 1.00-2.00 sec   112 MBytes   942 Mbits/sec  898/0         0     
1675K(1185)/1675K/14295(114) us  8234
[  1] 2.00-3.00 sec   112 MBytes   943 Mbits/sec  899/0         0     
1783K(1261)/1788K/15220(188) us  7742
[  1] 3.00-4.00 sec   112 MBytes   943 Mbits/sec  899/0         0     
1875K(1326)/1875K/16128(191) us  7306
[  1] 4.00-5.00 sec   112 MBytes   937 Mbits/sec  894/0         1     
1356K(959)/1356K/11166(273) us  10494
[  1] 5.00-6.00 sec   112 MBytes   942 Mbits/sec  898/0         0     
1450K(1026)/1450K/12334(159) us  9543
[  1] 6.00-7.00 sec   112 MBytes   942 Mbits/sec  898/0         0     
1524K(1078)/1524K/12974(165) us  9072
[  1] 7.00-8.00 sec   112 MBytes   943 Mbits/sec  899/0         0     
1576K(1115)/1576K/13436(112) us  8770
[  1] 8.00-9.00 sec   112 MBytes   942 Mbits/sec  898/0         0     
1610K(1139)/1610K/13789(157) us  8536
[  1] 9.00-10.00 sec   112 MBytes   942 Mbits/sec  898/0         0     
1638K(1159)/1638K/14001(104) us  8407
[  1] 0.00-10.03 sec  1.10 GBytes   940 Mbits/sec  8993/0        73      
   0K(0)/1638K/14017(120) us  8386

root@rpi5-35:~# iperf -s -i 1 -e
------------------------------------------------------------
Server listening on TCP port 5001 with pid 35260
Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
TCP congestion control default cubic
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.35%eth0 port 5001 connected with 192.168.1.103 
port 53192 (trip-times) (sock=4) (peer 2.2.0-rc) 
(icwnd/mss/irtt=14/1448/171) on 2024-03-25 16:54:02.796 (PDT)
[ ID] Interval        Transfer    Bandwidth    Burst Latency 
avg/min/max/stdev (cnt/size) inP NetPwr  Reads=Dist
[  1] 0.00-1.00 sec   112 MBytes   941 Mbits/sec  
13.100/1.372/34.740/4.362 ms (897/131165) 1.49 MByte 8981  
18597=18552:13:5:6:1:3:5:12
[  1] 1.00-2.00 sec   112 MBytes   941 Mbits/sec  
15.244/14.370/16.076/0.389 ms (898/131054) 1.71 MByte 7720  
19588=19588:0:0:0:0:0:0:0
[  1] 2.00-3.00 sec   112 MBytes   942 Mbits/sec  
16.325/15.569/17.066/0.318 ms (898/131061) 1.83 MByte 7209  
19568=19568:0:0:0:0:0:0:0
[  1] 3.00-4.00 sec   112 MBytes   941 Mbits/sec  
17.187/16.504/17.841/0.272 ms (898/131053) 1.93 MByte 6847  
19317=19317:0:0:0:0:0:0:0
[  1] 4.00-5.00 sec   112 MBytes   941 Mbits/sec  
17.979/14.707/34.753/1.359 ms (898/131050) 2.01 MByte 6546  
19044=19022:3:3:1:0:0:0:15
[  1] 5.00-6.00 sec   112 MBytes   941 Mbits/sec  
13.481/12.410/14.241/0.298 ms (898/131054) 1.51 MByte 8730  
18915=18910:2:1:1:1:0:0:0
[  1] 6.00-7.00 sec   112 MBytes   941 Mbits/sec  
14.220/13.581/14.831/0.245 ms (897/131199) 1.60 MByte 8276  
19495=19491:2:2:0:0:0:0:0
[  1] 7.00-8.00 sec   112 MBytes   941 Mbits/sec  
14.762/14.231/15.296/0.211 ms (898/131049) 1.66 MByte 7972  
19164=19161:2:1:0:0:0:0:0
[  1] 8.00-9.00 sec   112 MBytes   941 Mbits/sec  
15.144/14.693/15.589/0.180 ms (898/131052) 1.70 MByte 7771  
19270=19267:2:0:1:0:0:0:0
[  1] 9.00-10.00 sec   112 MBytes   942 Mbits/sec  
15.398/15.012/15.817/0.172 ms (898/131055) 1.73 MByte 7643  
18872=18866:5:0:1:0:0:0:0
[  1] 0.00-10.02 sec  1.10 GBytes   941 Mbits/sec  
15.285/1.372/34.753/2.068 ms (8993/131072) 1.68 MByte 7699  
192149=192061:29:12:10:2:3:5:27

Bob