MIME-Version: 1.0
From: Jonathan Lennox <jonathan.lennox42@gmail.com>
Date: Thu, 29 Oct 2026 15:57:15 -0400
Message-ID: 
 <CAKx+b+YvqRqFTkJwPuMwZ8Dq9-jg+kZoSmLhWwVvdo8JLsd1_Q@mail.gmail.com>
To: sbm@ietf.org
Content-Type: multipart/alternative; boundary="000000000000d1d226065f001642"
Message-ID-Hash: AP5WE6APMKA7YIV4RNPAGPEXQCZH2ZAG
Precedence: list
Subject: [sbm] Implementation experience and measurements: draft-cheshire-sbm-04
Archived-At: 
 <https://mailarchive.ietf.org/arch/msg/sbm/PPY8qQRVJS0EsnGORBKKlMAcMXQ>

--000000000000d1d226065f001642
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

*[Disclosure: This report was prepared with substantial AI assistance
(implementation, measurement runs, and drafting), working interactively
under my direction; I designed the experiments with it, challenged and
corrected its analyses, and have reviewed and stand behind every claim.]*

*Summary*

I implemented TCP_REPLENISH_TIME in the Linux TCP stack and measured it
against TCP_NOTSENT_LOWAT and against an unmodified sender, on one WAN path
(~172 ms RTT) at three bottleneck rates spanning three orders of magnitude
(100 kb/s, 2 Mb/s, ~86 Mb/s). The draft's central claim held up cleanly: a
single time-based threshold (15 ms) ran at full throughput at all three
rates with no retuning, with source queueing delay near the achievable
minimum at each rate (at 100 kb/s that minimum is a segment-time floor =E2=
=80=94
=C2=A76). Byte-based thresholds, by contrast, were wrong somewhere at every
setting =E2=80=94 at 100 kb/s there is no workable byte value at all =E2=80=
=94 and the
do-nothing default produced standing source queues from 1.3 s (fast path)
to over a minute (slow path).

Along the way I found that the mechanism needs one qualification the draft
does not currently state: the readiness rule must treat a Nagle-held
sub-MSS backlog as writable. Without that clause, the interaction between
time-based thresholds, Nagle's algorithm, and rate estimates derived from a
flow's own (application-limited) behavior locks a small-write sender at one
write per RTT; with it =E2=80=94 a one-line addition, and one XNU's
TCP_NOTSENT_LOWAT wakeup logic already ships =E2=80=94 the same workload ra=
n at
full link rate. Suggested draft text in =C2=A71, diagnosis and measurements=
 in
=C2=A76; I think the draft deserves a paragraph on it.

Suggestions first (=C2=A7=C2=A71=E2=80=932); the implementation, the measur=
ements, and the
diagnosis of the Nagle finding follow (=C2=A7=C2=A73=E2=80=936).

*1. Suggestions for the draft*

   1.

   *Application guidance (draft =C2=A77):* the draft currently says nothing
   about Nagle; a sentence recommending TCP_NODELAY alongside
   TCP_REPLENISH_TIME would be worthwhile. The reason is the latency goal
   itself, at low bitrates: under Nagle, data moves only in full segments, =
so
   the achievable source delay is floored at one MSS of serialization =E2=
=80=94
   MSS/rate, which at low rates dwarfs any reasonable REPLENISH_TIME settin=
g
   (measured at 100 kb/s: 174 ms median with Nagle on, versus 46 ms with
   TCP_NODELAY, at identical goodput). Delay-sensitive applications general=
ly
   set TCP_NODELAY already, but the draft stating it would make the depende=
ncy
   explicit: TCP_NODELAY is what actually delivers the configured latency w=
hen
   rate =C3=97 REPLENISH_TIME approaches one MSS.
   2.

   *Implementation guidance:* when Nagle is enabled, treat any sub-MSS
   unsent backlog as below-threshold (writable) =E2=80=94 the rule XNU alre=
ady ships
   in its TCP_NOTSENT_LOWAT wakeup logic (quoted in =C2=A76). A Nagle-held =
tail
   cannot drain until an ACK arrives *or the application supplies more data
   to complete the segment*, so inviting more data is precisely the action
   that reduces latency; it generalizes the "empty backlog is always writab=
le"
   rule and breaks the feedback loop diagnosed in =C2=A76. (The same reason=
ing
   probably argues for applying the rule while a socket is corked =E2=80=94=
 TCP_CORK
   holds sub-MSS data the same way =E2=80=94 but I have not measured that c=
ase.) The
   Nagle case is verified by measurement: with this rule in the kernel and
   Nagle left on, the 4 KiB workload that had been trapped at one write per
   RTT (=C2=A76) ran at full link rate with 23 ms median source delay =E2=
=80=94 matching
   the TCP_NODELAY result =E2=80=94 with no change to the well-behaved case=
s. Two
   boundaries are worth stating in the draft:
   - The rule floors the effective threshold at one MSS: below
      MSS/REPLENISH_TIME (~770 kb/s for 15 ms at MSS 1448,
proportionally higher
      for jumbo frames), the achieved advance notice degrades to MSS/rate. =
That
      is Nagle's floor, not the rule's =E2=80=94 under Nagle data moves onl=
y in full
      segments. Measured at 100 kb/s with 512 B writes: full rate at 149 ms
      median, consistent with the ~121 ms MSS/rate floor at this path's MSS=
 of
      1228.
      - The Nagle condition is essential; with TCP_NODELAY the rule must
      *not* apply. In a controlled comparison at the same 100 kb/s,
      inviting on sub-MSS backlog pooled writes behind a cwnd-blocked segme=
nt
      (123 ms median), while inviting only at backlog exhaustion let each w=
rite
      travel as its own small packet at identical goodput with 46 ms
median =E2=80=94 the
      per-packet serialization + ACK-clock floor, the best achievable for a
      saturating flow at that rate.

   Relatedly, draft =C2=A77.2's discussion of write sizing covers the
   oversized-write case well (the large-frame example: the loop
   self-regulates); the hazardous end turns out to be the opposite one =E2=
=80=94
   writes much smaller than rate =C3=97 REPLENISH_TIME, and especially smal=
ler than
   one MSS, which is the regime where the trap of =C2=A76 lives.
   3.

   *Implementation guidance =E2=80=94 notification only.* It may be worth s=
tating
   explicitly that TCP_REPLENISH_TIME implementations SHOULD NOT gate the
   write path: the threshold should affect readiness notification exclusive=
ly,
   never cause write()/send() to block or return short. Linux's
   TCP_NOTSENT_LOWAT gates sendmsg() as well, and that measurably distorted
   application behavior (=C2=A75).

*2. Smaller points*

   -

   *Units and range.* Microseconds as specified; suggest the draft state
   that 0 disables the mechanism and that implementations reject negative
   values.
   -

   *Estimator guidance.* Draft =C2=A75.2's list of factors is good. Concret=
ely,
   a delivery-rate sample (delivered bytes over a sample interval =E2=80=94=
 the
   estimator Linux exposes as tcpi_delivery_rate and BBR builds on) worked
   well and is app-limited-aware; cwnd/SRTT is an acceptable bootstrap befo=
re
   the first sample but overestimates drain rate for exactly the applicatio=
ns
   SBM targets.
   -

   *Answer to the author's note in draft =C2=A77.1 (Linux wakeup behavior).=
* The
   draft asks for confirmation of the "wake at 50% send-buffer occupancy"
   description. Close, but the actual Linux rule is: the socket reports
   writable when free space is at least *half of the queued amount* (
   sk_stream_min_wspace() / __sk_stream_is_writeable() in include/net/sock.=
h),
   i.e., the wakeup comes when occupancy falls to two-thirds of SO_SNDBUF, =
not
   half =E2=80=94 and "queued" counts sent-but-unacknowledged data as well =
as unsent.
   Our unmodified-sender baselines confirmed the =E2=85=94 equilibrium empi=
rically.
   The stated conclusion (large efficient writes instead of many small ones=
)
   is correct.
   -

   *Complementarity with AQM (draft =C2=A72).* Incidentally demonstrated: a=
n
   early misconfiguration left a ~6 s FIFO at the bottleneck, and the
   resulting multi-second RTT stalled ACK-clocked wakeups and confused loss
   recovery regardless of sender-side settings. SBM fixes the source queue
   only; the draft's positioning of SBM as a complement to AQM/L4S matched
   what we saw when both queues were present.
   -

   *Adoption note (draft =C2=A78.4).* The results support the draft's sugge=
stion
   that a modest default (e.g., 20 ms) would benefit unmodified event-loop
   applications: the mechanism cost nothing in throughput in these tests, a=
nd
   the failure mode of a too-generous value is simply the status quo.

*3. What was implemented*

A new TCP socket option, TCP_REPLENISH_TIME (microseconds, 0 =3D disabled,
the default), on Linux (patch against 7.x). Design choices, several of
which the draft could usefully standardize or at least discuss:

   -

   *Notification-only.* The option affects poll()/EPOLLOUT readiness and
   write-space wakeups exclusively; it never causes write()/send() to block=
 or
   return short. This follows draft =C2=A77.1 ("the absence of the kevent()=
 signal
   is not a prohibition"). This is a real behavioral difference from Linux
   TCP_NOTSENT_LOWAT, which also gates sendmsg() and therefore truncates la=
rge
   writes (see =C2=A75).
   -

   *Estimator.* The socket is reported writable when unsent_bytes <=3D
   drain_rate =C3=97 replenish_time. For drain_rate the implementation pref=
ers
   the kernel's delivery rate sampler (delivered bytes / sample interval, t=
he
   same estimator exposed as tcpi_delivery_rate); before the first rate sam=
ple
   exists it falls back to min(cwnd =C3=97 MSS, peer window) per SRTT. The
   delivery-rate sampler was chosen because congestion window over SRTT
   overestimates the drain rate of application-limited flows =E2=80=94 and =
SBM's
   target applications are application-limited by construction.
   -

   *Edge cases* (the draft is silent on all of these; suggested positions
   in parentheses): value 0 (disabled); no RTT/rate sample yet (report
   writable =E2=80=94 err toward inviting data rather than stalling); unsen=
t backlog
   exactly zero (always writable, regardless of the estimate); interaction
   with a simultaneously-set TCP_NOTSENT_LOWAT (both constraints must hold)=
;
   inheritance (a listener's setting is inherited by accepted sockets).
   -

   *Hysteresis.* Following existing Linux practice for write-readiness
   edges, the wakeup fires when the backlog drains to *half* the computed
   threshold, to avoid wakeup thrash. Observed advance notice was therefore
   roughly 0.5=E2=80=931.5=C3=97 the configured value, quantized by ACK arr=
ival. The
   draft's tolerance for imprecision (draft =C2=A76) comfortably covers thi=
s.

*4. Measurements*

Testbed: Linux sender (patched kernel) =E2=86=92 WAN (~172 ms RTT) =E2=86=
=92 cloud receiver
discarding to /dev/null. Bottleneck: receiver-side ingress HTB at 2 Mb/s
with a bounded queue (~200 ms) for the slow configuration; ~86 Mb/s
end-to-end otherwise. Sender: nonblocking epoll loop writing 64 KiB per
EPOLLOUT invitation; per-write sojourn-in-sndbuf measured with
SO_TIMESTAMPING (TX_SCHED, OPT_ID), i.e., "queued" =3D time from write() to
the last byte of that write leaving for the qdisc. Runs 35=E2=80=9345 s. CU=
BIC
congestion control.

Median / p95 source queueing delay, and achieved goodput:
Sender configuration 100 kb/s bottleneck 2 Mb/s bottleneck ~86 Mb/s path
Default (8 MB sndbuf) 85 s at t=3D90 s, still climbing (equilibrium =E2=89=
=88 8 min) 22.0
s / 23.4 s @ 1.81 M 1.31 s / 2.0 s @ 28.7 M
NOTSENT_LOWAT =3D 165 KB 18 s / 19.5 s standing 0.61 s / 0.76 s @ 1.92 M 11
ms / 40 ms @ 78.2 M
NOTSENT_LOWAT tuned to rate =3D 187 B (sub-MSS): goodput collapse to 15.7
kb/s (Nagle, =C2=A76) =3D 15 KB: ~0.2 s, goodput =E2=88=9225% (1.45 M) =3D =
165 KB: see row
above
REPLENISH_TIME =3D 15 ms 174 ms / 220 ms @ 84 kb/s (MSS floor, =C2=A76; 46 =
ms with
TCP_NODELAY) 0.28 s / 0.37 s @ 1.88 M 8.6 ms / 40 ms @ 86.6 M

Notes on reading the table:

   -

   The 100 kb/s column used 512 B writes (64 KiB writes would be 5.4 s of
   self-drain each at that rate); the other columns used 64 KiB. Goodput at
   100 kb/s was still converging at t=3D90 s, so the queue figures are the
   meaningful comparison in that column.
   -

   Measurement-validity check: the SO_TIMESTAMPING instrumentation itself
   wakes the sender via the error queue, which in principle could substitut=
e
   for missing EPOLLOUT wakeups and flatter the results. Control runs with
   timestamping disabled entirely produced identical goodput and inter-writ=
e
   cadence (within 1%) in all three write patterns, confirming the measured
   invitation timing comes from the genuine write-readiness wakeup path.
   -

   The 100 kb/s column is where byte thresholds run out of road entirely:
   the value tuned for the fast path parks 18 seconds of queue, and the val=
ue
   "correctly" tuned for 15 ms at this rate is smaller than one MSS, which
   triggers the Nagle interaction of =C2=A76 and collapses throughput ~6=C3=
=97. There is
   no good byte value at this rate.
   -

   The 0.28 s for REPLENISH_TIME at 2 Mb/s is almost entirely the drain
   time of the 64 KiB write itself (64 KiB =C3=97 8 =C3=B7 1.9 Mb/s =3D 0.2=
76 s); the
   mechanism added essentially zero *standing* queue. The advance-notice
   timing was directly observable: each invitation arrived 0=E2=80=9320 ms =
before the
   previous backlog's last byte left the host, i.e., right at the configure=
d
   15 ms.
   -

   165 KB is the "correct" byte threshold for 15 ms at 86 Mb/s. It is the
   best a byte-based tuner could do for the fast path, and it costs ~0.6 s =
of
   standing queue the moment the same connection runs at 2 Mb/s. Conversely
   the 15 KB value that suits 2 Mb/s cost ~25% of goodput there (sendmsg
   gating, =C2=A75) and is far too small for the fast path. The time-based =
option
   needed no retuning across the 40=C3=97 rate change =E2=80=94 this is the=
 draft's =C2=A75
   argument, observed directly.
   -

   Throughput cost of the mechanism: none observed. The REPLENISH_TIME run
   was the fastest run of the set on the unshaped path. (The
   default-configuration baseline at 28.7 Mb/s hit early losses and CUBIC
   recovers slowly at this RTT; the point is that keeping the source buffer
   nearly empty did not prevent full bandwidth utilization, including
   bandwidth *discovery* from slow start =E2=80=94 a concern I had about us=
ing the
   delivery-rate estimator, which did not materialize.)
   -

   Within a single 23 s connection on the unshaped path, the same 15 ms
   setting tracked the delivery rate from 1.5 Mb/s (early slow start) to ~9=
0
   Mb/s with correct invitations throughout.

*5. Observation: byte-threshold mechanisms that gate write() distort
application behavior*

Linux's TCP_NOTSENT_LOWAT gates sendmsg() as well as poll(): once the
unsent backlog crosses the threshold, a large write returns short (in our
runs, 1 MB writes returned ~58 KB, quantized by GSO skb size). Beyond the
API surprise, this had two measurable effects: the tuned 15 KB threshold
lost ~25% goodput at 2 Mb/s, and =E2=80=94 a measurement methodology note f=
or
anyone reproducing this =E2=80=94 sub-skb short writes coalesce into shared=
 skbs,
so SO_TIMESTAMPING OPT_ID reports collapse to one per merged skb
(documented Linux behavior), silently thinning per-write latency data. The
draft's notification-only stance avoids both problems =E2=80=94 hence sugge=
stion 3
in =C2=A71.

*6. Finding: the readiness rule needs a sub-MSS clause, or Nagle =E2=80=94 =
even
with Minshall's modification =E2=80=94 traps it at one write per RTT*

The finding, and the fix, up front: a REPLENISH_TIME (or byte-lowat) sender
making sub-MSS-tailed writes with Nagle enabled phase-locks at one write
per RTT. The cure is to treat any sub-MSS unsent backlog as writable while
Nagle is enabled =E2=80=94 a one-clause extension of the "empty backlog is =
always
writable" rule, which the patch now implements, which restored the trapped
workload to full link rate, and which turns out to be exactly what XNU has
long shipped in its TCP_NOTSENT_LOWAT wakeup logic (suggestion 2 in =C2=A71=
,
where the fix's measurements and boundary conditions are also stated). The
rest of this section is the evidence: the trap, why Minshall's modification
does not prevent it, and the macOS comparison.

Measured, on the implementation as first built =E2=80=94 before the sub-MSS=
 clause
was added: with 4 KiB writes on the 2 Mb/s / 172 ms path, a REPLENISH_TIME
sender locked at exactly one write per RTT (4096 B / 172 ms =E2=89=88 190 k=
b/s),
congestion window pinned near its initial value, flow permanently
application-limited. Setting TCP_NODELAY =E2=80=94 and changing nothing els=
e =E2=80=94 took
the same workload to full link rate with ~24 ms median source delay.

The mechanism of the trap is a three-way interaction:

   1.

   A 4 KiB write is a few full segments plus a sub-MSS tail (on this path,
   MSS 1228: three segments plus 412 B). Nagle =E2=80=94 in Linux, Minshall=
's
   modification of it =E2=80=94 holds the tail while a previously sent sub-=
MSS packet
   remains unacknowledged, so the unsent backlog never reaches zero: it par=
ks
   at the tail size.
   2.

   The writability threshold (rate =C3=97 REPLENISH_TIME in bytes) sits *be=
low*
   that tail, because the delivery-rate estimate is itself depressed by the
   resulting one-write-per-RTT behavior. The socket is therefore "correctly=
"
   unwritable by the estimate: positive feedback, and the flow never escape=
s.
   3.

   The ACK finally releases the tail, the backlog drains, the wakeup fires
   =E2=80=94 one invitation per RTT.

One might expect Minshall's modification to prevent this, since it exists
to avoid Nagle pathologies: it permits a sub-MSS packet whenever no
*previous* sub-MSS packet is unacknowledged, i.e., roughly one small packet
per RTT. It does not help here, because the loop consumes exactly that
allowance in phase-lock. The invitation fires at the moment the previous
tail departs =E2=80=94 that departure is what makes the backlog reach zero =
=E2=80=94 the
application writes immediately, and the new write's tail therefore always
finds the just-departed small packet still unacknowledged: held one RTT,
released by its ACK, whose processing issues the next invitation. Minshall
prevents small packets from *accumulating*; it cannot help a sender whose
write clock is phase-locked to the small-packet ACK clock. The measured
signature matches this to the byte: exactly one write per RTT, with the
unsent backlog parked at exactly 4096 =E2=88=92 3=C3=971228 =3D 412 B.

Nor is the trap specific to TCP_REPLENISH_TIME or to my patch: it
reproduces with plain TCP_NOTSENT_LOWAT (whose semantics the patch leaves
unchanged, and which behaves identically on stock Linux) whenever the byte
threshold is below one MSS. Measured: TCP_NOTSENT_LOWAT =3D 187 with 512 B
writes on the 100 kb/s path collapsed to 15.7 kb/s of ~90 available =E2=80=
=94 each
write held by Minshall=E2=80=93Nagle until the previous write's ACK. A time=
-based
threshold simply *inherits* this pre-existing lowat behavior whenever its
byte equivalent (rate =C3=97 REPLENISH_TIME) falls below one MSS =E2=80=94 =
and unlike a
byte threshold, it will cross that line silently as the rate drops, which
is why the guidance of =C2=A71 is worth stating.

Platform comparison: the lock does *not* reproduce on macOS. The same
workload (TCP_NOTSENT_LOWAT =3D 187, 512 B writes, kevent/EVFILT_WRITE, sam=
e
100 kb/s path, ~180 ms RTT) sustained ~79 kb/s =E2=80=94 effectively full r=
ate =E2=80=94
with sub-MSS data packets pipelined several per RTT. Reading XNU explains
why Darwin is immune, at two independent layers:

   -

   Its kevent readiness check, tcp_notsent_lowat_check(), explicitly
   reports the socket writable whenever the unsent backlog is nonzero but
   smaller than one MSS and Nagle is enabled ("When Nagle's algorithm is no=
t
   disabled, it is better to wakeup the client until there is atleast one
   maxseg of data to write"). The application is therefore invited until a
   full segment's worth pools, which then passes tcp_output's unconditional
   len >=3D maxseg send test.
   -

   Its Nagle implementation uses a different Minshall variant: a single
   flag (TF_MAXSEGSNT, "the last segment sent was full-sized") rather than
   Linux's "is a previously sent sub-MSS packet still unacknowledged"
   (snd_sml). In the 4 KiB-write pattern the trailing partial always direct=
ly
   follows full segments, so Darwin always sends it immediately, while Linu=
x's
   variant always finds the previous small packet unacked and always holds =
it.

The trap is therefore specific to stacks that combine Linux's form of
Minshall=E2=80=93Nagle with an unsent-byte readiness threshold lacking a su=
b-MSS
clause =E2=80=94 which makes the fix (=C2=A71, suggestion 2) less a proposa=
l of mine
than existing Apple practice, independently converged on.

The notification *machinery*, for its part, is blameless: Linux
re-evaluates write readiness on both ACK and transmit events (the latter
since 2022, commit 4bfe744ff164, "tcp: fix potential xmit stalls caused by
TCP_NOTSENT_LOWAT" =E2=80=94 the analogous wakeup problem, fixed long ago).
Readiness can only change at those points, since the backlog shrinks only
on transmit and the rate/RTT estimates update only on ACKs, and poll()
additionally re-evaluates synchronously. For the same reason, a timer-based
wakeup would not help: no input to the estimate changes between those
events.

*7. Availability*

Linux patch (new sockopt, ~75 lines against net/ipv4 + net/ipv6) and the
measurement tool (epoll + SO_TIMESTAMPING sender) available on request;
intending to post the patch as an RFC to netdev.


Jonathan Lennox

8x8 / Jitsi

jonathan.lennox42@gmail.com

--000000000000d1d226065f001642
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><p style=3D"color:rgb(17,17,17);margin:0px 0px 1em;font-si=
ze:18px"><i style=3D"font-size:14px;background-color:transparent">[Disclosu=
re: This report was prepared with substantial AI assistance (implementation=
, measurement runs, and drafting), working interactively under my direction=
; I designed the experiments with it, challenged and corrected its analyses=
, and have reviewed and stand behind every claim.]</i></p><p style=3D"color=
:rgb(17,17,17);margin:1.4em 0px 0.6em;font-size:15px"><b>Summary</b></p><p =
style=3D"color:rgb(17,17,17);font-size:14px;margin:0px 0px 1em">I implement=
ed TCP_REPLENISH_TIME in the Linux TCP stack and measured it against TCP_NO=
TSENT_LOWAT and against an unmodified sender, on one WAN path (~172 ms RTT)=
 at three bottleneck rates spanning three orders of magnitude (100 kb/s, 2 =
Mb/s, ~86 Mb/s). The draft&#39;s central claim held up cleanly: a single ti=
me-based threshold (15 ms) ran at full throughput at all three rates with n=
o retuning, with source queueing delay near the achievable minimum at each =
rate (at 100 kb/s that minimum is a segment-time floor =E2=80=94 =C2=A76). =
Byte-based thresholds, by contrast, were wrong somewhere at every setting =
=E2=80=94 at 100 kb/s there is no workable byte value at all =E2=80=94 and =
the do-nothing default produced standing source queues from 1.3 s (fast pat=
h) to over a minute (slow path).</p><p style=3D"color:rgb(17,17,17);font-si=
ze:14px;margin:0px 0px 1em">Along the way I found that the mechanism needs =
one qualification the draft does not currently state: the readiness rule mu=
st treat a Nagle-held sub-MSS backlog as writable. Without that clause, the=
 interaction between time-based thresholds, Nagle&#39;s algorithm, and rate=
 estimates derived from a flow&#39;s own (application-limited) behavior loc=
ks a small-write sender at one write per RTT; with it =E2=80=94 a one-line =
addition, and one XNU&#39;s TCP_NOTSENT_LOWAT wakeup logic already ships =
=E2=80=94 the same workload ran at full link rate. Suggested draft text in =
=C2=A71, diagnosis and measurements in =C2=A76; I think the draft deserves =
a paragraph on it.</p><p style=3D"color:rgb(17,17,17);font-size:14px;margin=
:0px 0px 1em">Suggestions first (=C2=A7=C2=A71=E2=80=932); the implementati=
on, the measurements, and the diagnosis of the Nagle finding follow (=C2=A7=
=C2=A73=E2=80=936).</p><p style=3D"color:rgb(17,17,17);margin:1.4em 0px 0.6=
em;font-size:15px"><b>1. Suggestions for the draft</b></p><ol style=3D"colo=
r:rgb(17,17,17);font-size:14px;margin:0px 0px 1em;padding-left:1.7em"><li s=
tyle=3D"margin:0px 0px 0.8em"><p style=3D"margin:0px 0px 0.5em"><b>Applicat=
ion guidance (draft =C2=A77):</b> the draft currently says nothing about Na=
gle; a sentence recommending TCP_NODELAY alongside TCP_REPLENISH_TIME would=
 be worthwhile. The reason is the latency goal itself, at low bitrates: und=
er Nagle, data moves only in full segments, so the achievable source delay =
is floored at one MSS of serialization =E2=80=94 MSS/rate, which at low rat=
es dwarfs any reasonable REPLENISH_TIME setting (measured at 100 kb/s: 174 =
ms median with Nagle on, versus 46 ms with TCP_NODELAY, at identical goodpu=
t). Delay-sensitive applications generally set TCP_NODELAY already, but the=
 draft stating it would make the dependency explicit: TCP_NODELAY is what a=
ctually delivers the configured latency when rate =C3=97 REPLENISH_TIME app=
roaches one MSS.</p></li><li style=3D"margin:0px 0px 0.8em"><p style=3D"mar=
gin:0px 0px 0.5em"><b>Implementation guidance:</b> when Nagle is enabled, t=
reat any sub-MSS unsent backlog as below-threshold (writable) =E2=80=94 the=
 rule XNU already ships in its TCP_NOTSENT_LOWAT wakeup logic (quoted in =
=C2=A76). A Nagle-held tail cannot drain until an ACK arrives <i>or the app=
lication supplies more data to complete the segment</i>, so inviting more d=
ata is precisely the action that reduces latency; it generalizes the &quot;=
empty backlog is always writable&quot; rule and breaks the feedback loop di=
agnosed in =C2=A76. (The same reasoning probably argues for applying the ru=
le while a socket is corked =E2=80=94 TCP_CORK holds sub-MSS data the same =
way =E2=80=94 but I have not measured that case.) The Nagle case is verifie=
d by measurement: with this rule in the kernel and Nagle left on, the 4 KiB=
 workload that had been trapped at one write per RTT (=C2=A76) ran at full =
link rate with 23 ms median source delay =E2=80=94 matching the TCP_NODELAY=
 result =E2=80=94 with no change to the well-behaved cases. Two boundaries =
are worth stating in the draft:</p><ul style=3D"margin:0.5em 0px 1em;paddin=
g-left:1.5em"><li style=3D"margin:0px 0px 0.5em">The rule floors the effect=
ive threshold at one MSS: below MSS/REPLENISH_TIME (~770 kb/s for 15 ms at =
MSS 1448, proportionally higher for jumbo frames), the achieved advance not=
ice degrades to MSS/rate. That is Nagle&#39;s floor, not the rule&#39;s =E2=
=80=94 under Nagle data moves only in full segments. Measured at 100 kb/s w=
ith 512 B writes: full rate at 149 ms median, consistent with the ~121 ms M=
SS/rate floor at this path&#39;s MSS of 1228.</li><li style=3D"margin:0px 0=
px 0.5em">The Nagle condition is essential; with TCP_NODELAY the rule must =
<i>not</i> apply. In a controlled comparison at the same 100 kb/s, inviting=
 on sub-MSS backlog pooled writes behind a cwnd-blocked segment (123 ms med=
ian), while inviting only at backlog exhaustion let each write travel as it=
s own small packet at identical goodput with 46 ms median =E2=80=94 the per=
-packet serialization + ACK-clock floor, the best achievable for a saturati=
ng flow at that rate.</li></ul><p style=3D"margin:0px 0px 0.5em">Relatedly,=
 draft =C2=A77.2&#39;s discussion of write sizing covers the oversized-writ=
e case well (the large-frame example: the loop self-regulates); the hazardo=
us end turns out to be the opposite one =E2=80=94 writes much smaller than =
rate =C3=97 REPLENISH_TIME, and especially smaller than one MSS, which is t=
he regime where the trap of =C2=A76 lives.</p></li><li style=3D"margin:0px =
0px 0.8em"><p style=3D"margin:0px 0px 0.5em"><b>Implementation guidance =E2=
=80=94 notification only.</b> It may be worth stating explicitly that TCP_R=
EPLENISH_TIME implementations SHOULD NOT gate the write path: the threshold=
 should affect readiness notification exclusively, never cause write()/send=
() to block or return short. Linux&#39;s TCP_NOTSENT_LOWAT gates sendmsg() =
as well, and that measurably distorted application behavior (=C2=A75).</p><=
/li></ol><p style=3D"color:rgb(17,17,17);margin:1.4em 0px 0.6em;font-size:1=
5px"><b>2. Smaller points</b></p><ul style=3D"color:rgb(17,17,17);font-size=
:14px;margin:0px 0px 1em;padding-left:1.7em"><li style=3D"margin:0px 0px 0.=
8em"><p style=3D"margin:0px 0px 0.5em"><b>Units and range.</b> Microseconds=
 as specified; suggest the draft state that 0 disables the mechanism and th=
at implementations reject negative values.</p></li><li style=3D"margin:0px =
0px 0.8em"><p style=3D"margin:0px 0px 0.5em"><b>Estimator guidance.</b> Dra=
ft =C2=A75.2&#39;s list of factors is good. Concretely, a delivery-rate sam=
ple (delivered bytes over a sample interval =E2=80=94 the estimator Linux e=
xposes as tcpi_delivery_rate and BBR builds on) worked well and is app-limi=
ted-aware; cwnd/SRTT is an acceptable bootstrap before the first sample but=
 overestimates drain rate for exactly the applications SBM targets.</p></li=
><li style=3D"margin:0px 0px 0.8em"><p style=3D"margin:0px 0px 0.5em"><b>An=
swer to the author&#39;s note in draft =C2=A77.1 (Linux wakeup behavior).</=
b> The draft asks for confirmation of the &quot;wake at 50% send-buffer occ=
upancy&quot; description. Close, but the actual Linux rule is: the socket r=
eports writable when free space is at least <i>half of the queued amount</i=
> (<code style=3D"font-family:Menlo,Consolas,monospace;font-size:12.6px;bac=
kground:rgb(242,242,242);padding:0px 2px">sk_stream_min_wspace()</code> / <=
code style=3D"font-family:Menlo,Consolas,monospace;font-size:12.6px;backgro=
und:rgb(242,242,242);padding:0px 2px">__sk_stream_is_writeable()</code> in =
<code style=3D"font-family:Menlo,Consolas,monospace;font-size:12.6px;backgr=
ound:rgb(242,242,242);padding:0px 2px">include/net/sock.h</code>), i.e., th=
e wakeup comes when occupancy falls to two-thirds of SO_SNDBUF, not half =
=E2=80=94 and &quot;queued&quot; counts sent-but-unacknowledged data as wel=
l as unsent. Our unmodified-sender baselines confirmed the =E2=85=94 equili=
brium empirically. The stated conclusion (large efficient writes instead of=
 many small ones) is correct.</p></li><li style=3D"margin:0px 0px 0.8em"><p=
 style=3D"margin:0px 0px 0.5em"><b>Complementarity with AQM (draft =C2=A72)=
.</b> Incidentally demonstrated: an early misconfiguration left a ~6 s FIFO=
 at the bottleneck, and the resulting multi-second RTT stalled ACK-clocked =
wakeups and confused loss recovery regardless of sender-side settings. SBM =
fixes the source queue only; the draft&#39;s positioning of SBM as a comple=
ment to AQM/L4S matched what we saw when both queues were present.</p></li>=
<li style=3D"margin:0px 0px 0.8em"><p style=3D"margin:0px 0px 0.5em"><b>Ado=
ption note (draft =C2=A78.4).</b> The results support the draft&#39;s sugge=
stion that a modest default (e.g., 20 ms) would benefit unmodified event-lo=
op applications: the mechanism cost nothing in throughput in these tests, a=
nd the failure mode of a too-generous value is simply the status quo.</p></=
li></ul><p style=3D"color:rgb(17,17,17);margin:1.4em 0px 0.6em;font-size:15=
px"><b>3. What was implemented</b></p><p style=3D"color:rgb(17,17,17);font-=
size:14px;margin:0px 0px 1em">A new TCP socket option, TCP_REPLENISH_TIME (=
microseconds, 0 =3D disabled, the default), on Linux (patch against 7.x). D=
esign choices, several of which the draft could usefully standardize or at =
least discuss:</p><ul style=3D"color:rgb(17,17,17);font-size:14px;margin:0p=
x 0px 1em;padding-left:1.7em"><li style=3D"margin:0px 0px 0.8em"><p style=
=3D"margin:0px 0px 0.5em"><b>Notification-only.</b> The option affects poll=
()/EPOLLOUT readiness and write-space wakeups exclusively; it never causes =
write()/send() to block or return short. This follows draft =C2=A77.1 (&quo=
t;the absence of the kevent() signal is not a prohibition&quot;). This is a=
 real behavioral difference from Linux TCP_NOTSENT_LOWAT, which also gates =
sendmsg() and therefore truncates large writes (see =C2=A75).</p></li><li s=
tyle=3D"margin:0px 0px 0.8em"><p style=3D"margin:0px 0px 0.5em"><b>Estimato=
r.</b> The socket is reported writable when <code style=3D"font-family:Menl=
o,Consolas,monospace;font-size:12.6px;background:rgb(242,242,242);padding:0=
px 2px">unsent_bytes &lt;=3D drain_rate =C3=97 replenish_time</code>. For d=
rain_rate the implementation prefers the kernel&#39;s delivery rate sampler=
 (delivered bytes / sample interval, the same estimator exposed as tcpi_del=
ivery_rate); before the first rate sample exists it falls back to min(cwnd =
=C3=97 MSS, peer window) per SRTT. The delivery-rate sampler was chosen bec=
ause congestion window over SRTT overestimates the drain rate of applicatio=
n-limited flows =E2=80=94 and SBM&#39;s target applications are application=
-limited by construction.</p></li><li style=3D"margin:0px 0px 0.8em"><p sty=
le=3D"margin:0px 0px 0.5em"><b>Edge cases</b> (the draft is silent on all o=
f these; suggested positions in parentheses): value 0 (disabled); no RTT/ra=
te sample yet (report writable =E2=80=94 err toward inviting data rather th=
an stalling); unsent backlog exactly zero (always writable, regardless of t=
he estimate); interaction with a simultaneously-set TCP_NOTSENT_LOWAT (both=
 constraints must hold); inheritance (a listener&#39;s setting is inherited=
 by accepted sockets).</p></li><li style=3D"margin:0px 0px 0.8em"><p style=
=3D"margin:0px 0px 0.5em"><b>Hysteresis.</b> Following existing Linux pract=
ice for write-readiness edges, the wakeup fires when the backlog drains to =
<i>half</i> the computed threshold, to avoid wakeup thrash. Observed advanc=
e notice was therefore roughly 0.5=E2=80=931.5=C3=97 the configured value, =
quantized by ACK arrival. The draft&#39;s tolerance for imprecision (draft =
=C2=A76) comfortably covers this.</p></li></ul><p style=3D"color:rgb(17,17,=
17);margin:1.4em 0px 0.6em;font-size:15px"><b>4. Measurements</b></p><p sty=
le=3D"color:rgb(17,17,17);font-size:14px;margin:0px 0px 1em">Testbed: Linux=
 sender (patched kernel) =E2=86=92 WAN (~172 ms RTT) =E2=86=92 cloud receiv=
er discarding to /dev/null. Bottleneck: receiver-side ingress HTB at 2 Mb/s=
 with a bounded queue (~200 ms) for the slow configuration; ~86 Mb/s end-to=
-end otherwise. Sender: nonblocking epoll loop writing 64 KiB per EPOLLOUT =
invitation; per-write sojourn-in-sndbuf measured with SO_TIMESTAMPING (TX_S=
CHED, OPT_ID), i.e., &quot;queued&quot; =3D time from write() to the last b=
yte of that write leaving for the qdisc. Runs 35=E2=80=9345 s. CUBIC conges=
tion control.</p><p style=3D"color:rgb(17,17,17);font-size:14px;margin:0px =
0px 1em">Median / p95 source queueing delay, and achieved goodput:</p><tabl=
e style=3D"border-collapse:collapse;margin:0px 0px 1em"><tbody><tr><th styl=
e=3D"border:1px solid rgb(153,153,153);padding:4px 8px;vertical-align:top;f=
ont-size:14.4px;background:rgb(238,238,238);text-align:left">Sender configu=
ration</th><th style=3D"border:1px solid rgb(153,153,153);padding:4px 8px;v=
ertical-align:top;font-size:14.4px;background:rgb(238,238,238);text-align:l=
eft">100 kb/s bottleneck</th><th style=3D"border:1px solid rgb(153,153,153)=
;padding:4px 8px;vertical-align:top;font-size:14.4px;background:rgb(238,238=
,238);text-align:left">2 Mb/s bottleneck</th><th style=3D"border:1px solid =
rgb(153,153,153);padding:4px 8px;vertical-align:top;font-size:14.4px;backgr=
ound:rgb(238,238,238);text-align:left">~86 Mb/s path</th></tr><tr><td style=
=3D"border:1px solid rgb(153,153,153);padding:4px 8px;vertical-align:top;fo=
nt-size:14.4px">Default (8 MB sndbuf)</td><td style=3D"border:1px solid rgb=
(153,153,153);padding:4px 8px;vertical-align:top;font-size:14.4px">85 s at =
t=3D90 s, still climbing (equilibrium =E2=89=88 8 min)</td><td style=3D"bor=
der:1px solid rgb(153,153,153);padding:4px 8px;vertical-align:top;font-size=
:14.4px">22.0 s / 23.4 s @ 1.81 M</td><td style=3D"border:1px solid rgb(153=
,153,153);padding:4px 8px;vertical-align:top;font-size:14.4px">1.31 s / 2.0=
 s @ 28.7 M</td></tr><tr><td style=3D"border:1px solid rgb(153,153,153);pad=
ding:4px 8px;vertical-align:top;font-size:14.4px">NOTSENT_LOWAT =3D 165 KB<=
/td><td style=3D"border:1px solid rgb(153,153,153);padding:4px 8px;vertical=
-align:top;font-size:14.4px">18 s / 19.5 s standing</td><td style=3D"border=
:1px solid rgb(153,153,153);padding:4px 8px;vertical-align:top;font-size:14=
.4px">0.61 s / 0.76 s @ 1.92 M</td><td style=3D"border:1px solid rgb(153,15=
3,153);padding:4px 8px;vertical-align:top;font-size:14.4px">11 ms / 40 ms @=
 78.2 M</td></tr><tr><td style=3D"border:1px solid rgb(153,153,153);padding=
:4px 8px;vertical-align:top;font-size:14.4px">NOTSENT_LOWAT tuned to rate</=
td><td style=3D"border:1px solid rgb(153,153,153);padding:4px 8px;vertical-=
align:top;font-size:14.4px">=3D 187 B (sub-MSS): goodput collapse to 15.7 k=
b/s (Nagle, =C2=A76)</td><td style=3D"border:1px solid rgb(153,153,153);pad=
ding:4px 8px;vertical-align:top;font-size:14.4px">=3D 15 KB: ~0.2 s, goodpu=
t =E2=88=9225% (1.45 M)</td><td style=3D"border:1px solid rgb(153,153,153);=
padding:4px 8px;vertical-align:top;font-size:14.4px">=3D 165 KB: see row ab=
ove</td></tr><tr><td style=3D"border:1px solid rgb(153,153,153);padding:4px=
 8px;vertical-align:top;font-size:14.4px">REPLENISH_TIME =3D 15 ms</td><td =
style=3D"border:1px solid rgb(153,153,153);padding:4px 8px;vertical-align:t=
op;font-size:14.4px">174 ms / 220 ms @ 84 kb/s (MSS floor, =C2=A76; 46 ms w=
ith TCP_NODELAY)</td><td style=3D"border:1px solid rgb(153,153,153);padding=
:4px 8px;vertical-align:top;font-size:14.4px">0.28 s / 0.37 s @ 1.88 M</td>=
<td style=3D"border:1px solid rgb(153,153,153);padding:4px 8px;vertical-ali=
gn:top;font-size:14.4px">8.6 ms / 40 ms @ 86.6 M</td></tr></tbody></table><=
p style=3D"color:rgb(17,17,17);font-size:14px;margin:0px 0px 1em">Notes on =
reading the table:</p><ul style=3D"color:rgb(17,17,17);font-size:14px;margi=
n:0px 0px 1em;padding-left:1.7em"><li style=3D"margin:0px 0px 0.8em"><p sty=
le=3D"margin:0px 0px 0.5em">The 100 kb/s column used 512 B writes (64 KiB w=
rites would be 5.4 s of self-drain each at that rate); the other columns us=
ed 64 KiB. Goodput at 100 kb/s was still converging at t=3D90 s, so the que=
ue figures are the meaningful comparison in that column.</p></li><li style=
=3D"margin:0px 0px 0.8em"><p style=3D"margin:0px 0px 0.5em">Measurement-val=
idity check: the SO_TIMESTAMPING instrumentation itself wakes the sender vi=
a the error queue, which in principle could substitute for missing EPOLLOUT=
 wakeups and flatter the results. Control runs with timestamping disabled e=
ntirely produced identical goodput and inter-write cadence (within 1%) in a=
ll three write patterns, confirming the measured invitation timing comes fr=
om the genuine write-readiness wakeup path.</p></li><li style=3D"margin:0px=
 0px 0.8em"><p style=3D"margin:0px 0px 0.5em">The 100 kb/s column is where =
byte thresholds run out of road entirely: the value tuned for the fast path=
 parks 18 seconds of queue, and the value &quot;correctly&quot; tuned for 1=
5 ms at this rate is smaller than one MSS, which triggers the Nagle interac=
tion of =C2=A76 and collapses throughput ~6=C3=97. There is no good byte va=
lue at this rate.</p></li><li style=3D"margin:0px 0px 0.8em"><p style=3D"ma=
rgin:0px 0px 0.5em">The 0.28 s for REPLENISH_TIME at 2 Mb/s is almost entir=
ely the drain time of the 64 KiB write itself (64 KiB =C3=97 8 =C3=B7 1.9 M=
b/s =3D 0.276 s); the mechanism added essentially zero <i>standing</i> queu=
e. The advance-notice timing was directly observable: each invitation arriv=
ed 0=E2=80=9320 ms before the previous backlog&#39;s last byte left the hos=
t, i.e., right at the configured 15 ms.</p></li><li style=3D"margin:0px 0px=
 0.8em"><p style=3D"margin:0px 0px 0.5em">165 KB is the &quot;correct&quot;=
 byte threshold for 15 ms at 86 Mb/s. It is the best a byte-based tuner cou=
ld do for the fast path, and it costs ~0.6 s of standing queue the moment t=
he same connection runs at 2 Mb/s. Conversely the 15 KB value that suits 2 =
Mb/s cost ~25% of goodput there (sendmsg gating, =C2=A75) and is far too sm=
all for the fast path. The time-based option needed no retuning across the =
40=C3=97 rate change =E2=80=94 this is the draft&#39;s =C2=A75 argument, ob=
served directly.</p></li><li style=3D"margin:0px 0px 0.8em"><p style=3D"mar=
gin:0px 0px 0.5em">Throughput cost of the mechanism: none observed. The REP=
LENISH_TIME run was the fastest run of the set on the unshaped path. (The d=
efault-configuration baseline at 28.7 Mb/s hit early losses and CUBIC recov=
ers slowly at this RTT; the point is that keeping the source buffer nearly =
empty did not prevent full bandwidth utilization, including bandwidth <i>di=
scovery</i> from slow start =E2=80=94 a concern I had about using the deliv=
ery-rate estimator, which did not materialize.)</p></li><li style=3D"margin=
:0px 0px 0.8em"><p style=3D"margin:0px 0px 0.5em">Within a single 23 s conn=
ection on the unshaped path, the same 15 ms setting tracked the delivery ra=
te from 1.5 Mb/s (early slow start) to ~90 Mb/s with correct invitations th=
roughout.</p></li></ul><p style=3D"color:rgb(17,17,17);margin:1.4em 0px 0.6=
em;font-size:15px"><b>5. Observation: byte-threshold mechanisms that gate w=
rite() distort application behavior</b></p><p style=3D"color:rgb(17,17,17);=
font-size:14px;margin:0px 0px 1em">Linux&#39;s TCP_NOTSENT_LOWAT gates send=
msg() as well as poll(): once the unsent backlog crosses the threshold, a l=
arge write returns short (in our runs, 1 MB writes returned ~58 KB, quantiz=
ed by GSO skb size). Beyond the API surprise, this had two measurable effec=
ts: the tuned 15 KB threshold lost ~25% goodput at 2 Mb/s, and =E2=80=94 a =
measurement methodology note for anyone reproducing this =E2=80=94 sub-skb =
short writes coalesce into shared skbs, so SO_TIMESTAMPING OPT_ID reports c=
ollapse to one per merged skb (documented Linux behavior), silently thinnin=
g per-write latency data. The draft&#39;s notification-only stance avoids b=
oth problems =E2=80=94 hence suggestion 3 in =C2=A71.</p><p style=3D"color:=
rgb(17,17,17);margin:1.4em 0px 0.6em;font-size:15px"><b>6. Finding: the rea=
diness rule needs a sub-MSS clause, or Nagle =E2=80=94 even with Minshall&#=
39;s modification =E2=80=94 traps it at one write per RTT</b></p><p style=
=3D"color:rgb(17,17,17);font-size:14px;margin:0px 0px 1em">The finding, and=
 the fix, up front: a REPLENISH_TIME (or byte-lowat) sender making sub-MSS-=
tailed writes with Nagle enabled phase-locks at one write per RTT. The cure=
 is to treat any sub-MSS unsent backlog as writable while Nagle is enabled =
=E2=80=94 a one-clause extension of the &quot;empty backlog is always writa=
ble&quot; rule, which the patch now implements, which restored the trapped =
workload to full link rate, and which turns out to be exactly what XNU has =
long shipped in its TCP_NOTSENT_LOWAT wakeup logic (suggestion 2 in =C2=A71=
, where the fix&#39;s measurements and boundary conditions are also stated)=
. The rest of this section is the evidence: the trap, why Minshall&#39;s mo=
dification does not prevent it, and the macOS comparison.</p><p style=3D"co=
lor:rgb(17,17,17);font-size:14px;margin:0px 0px 1em">Measured, on the imple=
mentation as first built =E2=80=94 before the sub-MSS clause was added: wit=
h 4 KiB writes on the 2 Mb/s / 172 ms path, a REPLENISH_TIME sender locked =
at exactly one write per RTT (4096 B / 172 ms =E2=89=88 190 kb/s), congesti=
on window pinned near its initial value, flow permanently application-limit=
ed. Setting TCP_NODELAY =E2=80=94 and changing nothing else =E2=80=94 took =
the same workload to full link rate with ~24 ms median source delay.</p><p =
style=3D"color:rgb(17,17,17);font-size:14px;margin:0px 0px 1em">The mechani=
sm of the trap is a three-way interaction:</p><ol style=3D"color:rgb(17,17,=
17);font-size:14px;margin:0px 0px 1em;padding-left:1.7em"><li style=3D"marg=
in:0px 0px 0.8em"><p style=3D"margin:0px 0px 0.5em">A 4 KiB write is a few =
full segments plus a sub-MSS tail (on this path, MSS 1228: three segments p=
lus 412 B). Nagle =E2=80=94 in Linux, Minshall&#39;s modification of it =E2=
=80=94 holds the tail while a previously sent sub-MSS packet remains unackn=
owledged, so the unsent backlog never reaches zero: it parks at the tail si=
ze.</p></li><li style=3D"margin:0px 0px 0.8em"><p style=3D"margin:0px 0px 0=
.5em">The writability threshold (rate =C3=97 REPLENISH_TIME in bytes) sits =
<i>below</i> that tail, because the delivery-rate estimate is itself depres=
sed by the resulting one-write-per-RTT behavior. The socket is therefore &q=
uot;correctly&quot; unwritable by the estimate: positive feedback, and the =
flow never escapes.</p></li><li style=3D"margin:0px 0px 0.8em"><p style=3D"=
margin:0px 0px 0.5em">The ACK finally releases the tail, the backlog drains=
, the wakeup fires =E2=80=94 one invitation per RTT.</p></li></ol><p style=
=3D"color:rgb(17,17,17);font-size:14px;margin:0px 0px 1em">One might expect=
 Minshall&#39;s modification to prevent this, since it exists to avoid Nagl=
e pathologies: it permits a sub-MSS packet whenever no <i>previous</i> sub-=
MSS packet is unacknowledged, i.e., roughly one small packet per RTT. It do=
es not help here, because the loop consumes exactly that allowance in phase=
-lock. The invitation fires at the moment the previous tail departs =E2=80=
=94 that departure is what makes the backlog reach zero =E2=80=94 the appli=
cation writes immediately, and the new write&#39;s tail therefore always fi=
nds the just-departed small packet still unacknowledged: held one RTT, rele=
ased by its ACK, whose processing issues the next invitation. Minshall prev=
ents small packets from <i>accumulating</i>; it cannot help a sender whose =
write clock is phase-locked to the small-packet ACK clock. The measured sig=
nature matches this to the byte: exactly one write per RTT, with the unsent=
 backlog parked at exactly 4096 =E2=88=92 3=C3=971228 =3D 412 B.</p><p styl=
e=3D"color:rgb(17,17,17);font-size:14px;margin:0px 0px 1em">Nor is the trap=
 specific to TCP_REPLENISH_TIME or to my patch: it reproduces with plain TC=
P_NOTSENT_LOWAT (whose semantics the patch leaves unchanged, and which beha=
ves identically on stock Linux) whenever the byte threshold is below one MS=
S. Measured: TCP_NOTSENT_LOWAT =3D 187 with 512 B writes on the 100 kb/s pa=
th collapsed to 15.7 kb/s of ~90 available =E2=80=94 each write held by Min=
shall=E2=80=93Nagle until the previous write&#39;s ACK. A time-based thresh=
old simply <i>inherits</i> this pre-existing lowat behavior whenever its by=
te equivalent (rate =C3=97 REPLENISH_TIME) falls below one MSS =E2=80=94 an=
d unlike a byte threshold, it will cross that line silently as the rate dro=
ps, which is why the guidance of =C2=A71 is worth stating.</p><p style=3D"c=
olor:rgb(17,17,17);font-size:14px;margin:0px 0px 1em">Platform comparison: =
the lock does <i>not</i> reproduce on macOS. The same workload (TCP_NOTSENT=
_LOWAT =3D 187, 512 B writes, kevent/EVFILT_WRITE, same 100 kb/s path, ~180=
 ms RTT) sustained ~79 kb/s =E2=80=94 effectively full rate =E2=80=94 with =
sub-MSS data packets pipelined several per RTT. Reading XNU explains why Da=
rwin is immune, at two independent layers:</p><ul style=3D"color:rgb(17,17,=
17);font-size:14px;margin:0px 0px 1em;padding-left:1.7em"><li style=3D"marg=
in:0px 0px 0.8em"><p style=3D"margin:0px 0px 0.5em">Its kevent readiness ch=
eck, tcp_notsent_lowat_check(), explicitly reports the socket writable when=
ever the unsent backlog is nonzero but smaller than one MSS and Nagle is en=
abled (&quot;When Nagle&#39;s algorithm is not disabled, it is better to wa=
keup the client until there is atleast one maxseg of data to write&quot;). =
The application is therefore invited until a full segment&#39;s worth pools=
, which then passes tcp_output&#39;s unconditional len &gt;=3D maxseg send =
test.</p></li><li style=3D"margin:0px 0px 0.8em"><p style=3D"margin:0px 0px=
 0.5em">Its Nagle implementation uses a different Minshall variant: a singl=
e flag (TF_MAXSEGSNT, &quot;the last segment sent was full-sized&quot;) rat=
her than Linux&#39;s &quot;is a previously sent sub-MSS packet still unackn=
owledged&quot; (snd_sml). In the 4 KiB-write pattern the trailing partial a=
lways directly follows full segments, so Darwin always sends it immediately=
, while Linux&#39;s variant always finds the previous small packet unacked =
and always holds it.</p></li></ul><p style=3D"color:rgb(17,17,17);font-size=
:14px;margin:0px 0px 1em">The trap is therefore specific to stacks that com=
bine Linux&#39;s form of Minshall=E2=80=93Nagle with an unsent-byte readine=
ss threshold lacking a sub-MSS clause =E2=80=94 which makes the fix (=C2=A7=
1, suggestion 2) less a proposal of mine than existing Apple practice, inde=
pendently converged on.</p><p style=3D"color:rgb(17,17,17);font-size:14px;m=
argin:0px 0px 1em">The notification <i>machinery</i>, for its part, is blam=
eless: Linux re-evaluates write readiness on both ACK and transmit events (=
the latter since 2022, commit 4bfe744ff164, &quot;tcp: fix potential xmit s=
talls caused by TCP_NOTSENT_LOWAT&quot; =E2=80=94 the analogous wakeup prob=
lem, fixed long ago). Readiness can only change at those points, since the =
backlog shrinks only on transmit and the rate/RTT estimates update only on =
ACKs, and poll() additionally re-evaluates synchronously. For the same reas=
on, a timer-based wakeup would not help: no input to the estimate changes b=
etween those events.</p><p style=3D"color:rgb(17,17,17);margin:1.4em 0px 0.=
6em;font-size:15px"><b>7. Availability</b></p><p style=3D"color:rgb(17,17,1=
7);font-size:14px;margin:0px 0px 1em">Linux patch (new sockopt, ~75 lines a=
gainst net/ipv4 + net/ipv6) and the measurement tool (epoll + SO_TIMESTAMPI=
NG sender) available on request; intending to post the patch as an RFC to n=
etdev.</p><p style=3D"color:rgb(17,17,17);font-size:14px;margin:0px 0px 1em=
"><br></p><p style=3D"color:rgb(17,17,17);font-size:14px;margin:0px 0px 1em=
"><span style=3D"background-color:transparent">Jonathan Lennox</span></p><p=
 style=3D"color:rgb(17,17,17);font-size:14px;margin:0px 0px 1em">8x8 / Jits=
i</p><p style=3D"color:rgb(17,17,17);font-size:14px;margin:0px 0px 1em"><a =
href=3D"mailto:jonathan.lennox42@gmail.com">jonathan.lennox42@gmail.com</a>=
</p></div>

--000000000000d1d226065f001642--