[sbm] Implementation experience and measurements: draft-cheshire-sbm-04

[Jonathan Lennox <[e-mail]>]

*[Disclosure: This report was prepared with substantial AI assistance
(implementation, measurement runs, and drafting), working interactively
under my direction; I designed the experiments with it, challenged and
corrected its analyses, and have reviewed and stand behind every claim.]*

*Summary*

I implemented TCP_REPLENISH_TIME in the Linux TCP stack and measured it
against TCP_NOTSENT_LOWAT and against an unmodified sender, on one WAN path
(~172 ms RTT) at three bottleneck rates spanning three orders of magnitude
(100 kb/s, 2 Mb/s, ~86 Mb/s). The draft's central claim held up cleanly: a
single time-based threshold (15 ms) ran at full throughput at all three
rates with no retuning, with source queueing delay near the achievable
minimum at each rate (at 100 kb/s that minimum is a segment-time floor —
§6). Byte-based thresholds, by contrast, were wrong somewhere at every
setting — at 100 kb/s there is no workable byte value at all — and the
do-nothing default produced standing source queues from 1.3 s (fast path)
to over a minute (slow path).

Along the way I found that the mechanism needs one qualification the draft
does not currently state: the readiness rule must treat a Nagle-held
sub-MSS backlog as writable. Without that clause, the interaction between
time-based thresholds, Nagle's algorithm, and rate estimates derived from a
flow's own (application-limited) behavior locks a small-write sender at one
write per RTT; with it — a one-line addition, and one XNU's
TCP_NOTSENT_LOWAT wakeup logic already ships — the same workload ran at
full link rate. Suggested draft text in §1, diagnosis and measurements in
§6; I think the draft deserves a paragraph on it.

Suggestions first (§§1–2); the implementation, the measurements, and the
diagnosis of the Nagle finding follow (§§3–6).

*1. Suggestions for the draft*

   1.

   *Application guidance (draft §7):* the draft currently says nothing
   about Nagle; a sentence recommending TCP_NODELAY alongside
   TCP_REPLENISH_TIME would be worthwhile. The reason is the latency goal
   itself, at low bitrates: under Nagle, data moves only in full segments, so
   the achievable source delay is floored at one MSS of serialization —
   MSS/rate, which at low rates dwarfs any reasonable REPLENISH_TIME setting
   (measured at 100 kb/s: 174 ms median with Nagle on, versus 46 ms with
   TCP_NODELAY, at identical goodput). Delay-sensitive applications generally
   set TCP_NODELAY already, but the draft stating it would make the dependency
   explicit: TCP_NODELAY is what actually delivers the configured latency when
   rate × REPLENISH_TIME approaches one MSS.
   2.

   *Implementation guidance:* when Nagle is enabled, treat any sub-MSS
   unsent backlog as below-threshold (writable) — the rule XNU already ships
   in its TCP_NOTSENT_LOWAT wakeup logic (quoted in §6). A Nagle-held tail
   cannot drain until an ACK arrives *or the application supplies more data
   to complete the segment*, so inviting more data is precisely the action
   that reduces latency; it generalizes the "empty backlog is always writable"
   rule and breaks the feedback loop diagnosed in §6. (The same reasoning
   probably argues for applying the rule while a socket is corked — TCP_CORK
   holds sub-MSS data the same way — but I have not measured that case.) The
   Nagle case is verified by measurement: with this rule in the kernel and
   Nagle left on, the 4 KiB workload that had been trapped at one write per
   RTT (§6) ran at full link rate with 23 ms median source delay — matching
   the TCP_NODELAY result — with no change to the well-behaved cases. Two
   boundaries are worth stating in the draft:
   - The rule floors the effective threshold at one MSS: below
      MSS/REPLENISH_TIME (~770 kb/s for 15 ms at MSS 1448,
proportionally higher
      for jumbo frames), the achieved advance notice degrades to MSS/rate. That
      is Nagle's floor, not the rule's — under Nagle data moves only in full
      segments. Measured at 100 kb/s with 512 B writes: full rate at 149 ms
      median, consistent with the ~121 ms MSS/rate floor at this path's MSS of
      1228.
      - The Nagle condition is essential; with TCP_NODELAY the rule must
      *not* apply. In a controlled comparison at the same 100 kb/s,
      inviting on sub-MSS backlog pooled writes behind a cwnd-blocked segment
      (123 ms median), while inviting only at backlog exhaustion let each write
      travel as its own small packet at identical goodput with 46 ms
median — the
      per-packet serialization + ACK-clock floor, the best achievable for a
      saturating flow at that rate.

   Relatedly, draft §7.2's discussion of write sizing covers the
   oversized-write case well (the large-frame example: the loop
   self-regulates); the hazardous end turns out to be the opposite one —
   writes much smaller than rate × REPLENISH_TIME, and especially smaller than
   one MSS, which is the regime where the trap of §6 lives.
   3.

   *Implementation guidance — notification only.* It may be worth stating
   explicitly that TCP_REPLENISH_TIME implementations SHOULD NOT gate the
   write path: the threshold should affect readiness notification exclusively,
   never cause write()/send() to block or return short. Linux's
   TCP_NOTSENT_LOWAT gates sendmsg() as well, and that measurably distorted
   application behavior (§5).

*2. Smaller points*

   -

   *Units and range.* Microseconds as specified; suggest the draft state
   that 0 disables the mechanism and that implementations reject negative
   values.
   -

   *Estimator guidance.* Draft §5.2's list of factors is good. Concretely,
   a delivery-rate sample (delivered bytes over a sample interval — the
   estimator Linux exposes as tcpi_delivery_rate and BBR builds on) worked
   well and is app-limited-aware; cwnd/SRTT is an acceptable bootstrap before
   the first sample but overestimates drain rate for exactly the applications
   SBM targets.
   -

   *Answer to the author's note in draft §7.1 (Linux wakeup behavior).* The
   draft asks for confirmation of the "wake at 50% send-buffer occupancy"
   description. Close, but the actual Linux rule is: the socket reports
   writable when free space is at least *half of the queued amount* (
   sk_stream_min_wspace() / __sk_stream_is_writeable() in include/net/sock.h),
   i.e., the wakeup comes when occupancy falls to two-thirds of SO_SNDBUF, not
   half — and "queued" counts sent-but-unacknowledged data as well as unsent.
   Our unmodified-sender baselines confirmed the ⅔ equilibrium empirically.
   The stated conclusion (large efficient writes instead of many small ones)
   is correct.
   -

   *Complementarity with AQM (draft §2).* Incidentally demonstrated: an
   early misconfiguration left a ~6 s FIFO at the bottleneck, and the
   resulting multi-second RTT stalled ACK-clocked wakeups and confused loss
   recovery regardless of sender-side settings. SBM fixes the source queue
   only; the draft's positioning of SBM as a complement to AQM/L4S matched
   what we saw when both queues were present.
   -

   *Adoption note (draft §8.4).* The results support the draft's suggestion
   that a modest default (e.g., 20 ms) would benefit unmodified event-loop
   applications: the mechanism cost nothing in throughput in these tests, and
   the failure mode of a too-generous value is simply the status quo.

*3. What was implemented*

A new TCP socket option, TCP_REPLENISH_TIME (microseconds, 0 = disabled,
the default), on Linux (patch against 7.x). Design choices, several of
which the draft could usefully standardize or at least discuss:

   -

   *Notification-only.* The option affects poll()/EPOLLOUT readiness and
   write-space wakeups exclusively; it never causes write()/send() to block or
   return short. This follows draft §7.1 ("the absence of the kevent() signal
   is not a prohibition"). This is a real behavioral difference from Linux
   TCP_NOTSENT_LOWAT, which also gates sendmsg() and therefore truncates large
   writes (see §5).
   -

   *Estimator.* The socket is reported writable when unsent_bytes <=
   drain_rate × replenish_time. For drain_rate the implementation prefers
   the kernel's delivery rate sampler (delivered bytes / sample interval, the
   same estimator exposed as tcpi_delivery_rate); before the first rate sample
   exists it falls back to min(cwnd × MSS, peer window) per SRTT. The
   delivery-rate sampler was chosen because congestion window over SRTT
   overestimates the drain rate of application-limited flows — and SBM's
   target applications are application-limited by construction.
   -

   *Edge cases* (the draft is silent on all of these; suggested positions
   in parentheses): value 0 (disabled); no RTT/rate sample yet (report
   writable — err toward inviting data rather than stalling); unsent backlog
   exactly zero (always writable, regardless of the estimate); interaction
   with a simultaneously-set TCP_NOTSENT_LOWAT (both constraints must hold);
   inheritance (a listener's setting is inherited by accepted sockets).
   -

   *Hysteresis.* Following existing Linux practice for write-readiness
   edges, the wakeup fires when the backlog drains to *half* the computed
   threshold, to avoid wakeup thrash. Observed advance notice was therefore
   roughly 0.5–1.5× the configured value, quantized by ACK arrival. The
   draft's tolerance for imprecision (draft §6) comfortably covers this.

*4. Measurements*

Testbed: Linux sender (patched kernel) → WAN (~172 ms RTT) → cloud receiver
discarding to /dev/null. Bottleneck: receiver-side ingress HTB at 2 Mb/s
with a bounded queue (~200 ms) for the slow configuration; ~86 Mb/s
end-to-end otherwise. Sender: nonblocking epoll loop writing 64 KiB per
EPOLLOUT invitation; per-write sojourn-in-sndbuf measured with
SO_TIMESTAMPING (TX_SCHED, OPT_ID), i.e., "queued" = time from write() to
the last byte of that write leaving for the qdisc. Runs 35–45 s. CUBIC
congestion control.

Median / p95 source queueing delay, and achieved goodput:
Sender configuration 100 kb/s bottleneck 2 Mb/s bottleneck ~86 Mb/s path
Default (8 MB sndbuf) 85 s at t=90 s, still climbing (equilibrium ≈ 8 min) 22.0
s / 23.4 s @ 1.81 M 1.31 s / 2.0 s @ 28.7 M
NOTSENT_LOWAT = 165 KB 18 s / 19.5 s standing 0.61 s / 0.76 s @ 1.92 M 11
ms / 40 ms @ 78.2 M
NOTSENT_LOWAT tuned to rate = 187 B (sub-MSS): goodput collapse to 15.7
kb/s (Nagle, §6) = 15 KB: ~0.2 s, goodput −25% (1.45 M) = 165 KB: see row
above
REPLENISH_TIME = 15 ms 174 ms / 220 ms @ 84 kb/s (MSS floor, §6; 46 ms with
TCP_NODELAY) 0.28 s / 0.37 s @ 1.88 M 8.6 ms / 40 ms @ 86.6 M

Notes on reading the table:

   -

   The 100 kb/s column used 512 B writes (64 KiB writes would be 5.4 s of
   self-drain each at that rate); the other columns used 64 KiB. Goodput at
   100 kb/s was still converging at t=90 s, so the queue figures are the
   meaningful comparison in that column.
   -

   Measurement-validity check: the SO_TIMESTAMPING instrumentation itself
   wakes the sender via the error queue, which in principle could substitute
   for missing EPOLLOUT wakeups and flatter the results. Control runs with
   timestamping disabled entirely produced identical goodput and inter-write
   cadence (within 1%) in all three write patterns, confirming the measured
   invitation timing comes from the genuine write-readiness wakeup path.
   -

   The 100 kb/s column is where byte thresholds run out of road entirely:
   the value tuned for the fast path parks 18 seconds of queue, and the value
   "correctly" tuned for 15 ms at this rate is smaller than one MSS, which
   triggers the Nagle interaction of §6 and collapses throughput ~6×. There is
   no good byte value at this rate.
   -

   The 0.28 s for REPLENISH_TIME at 2 Mb/s is almost entirely the drain
   time of the 64 KiB write itself (64 KiB × 8 ÷ 1.9 Mb/s = 0.276 s); the
   mechanism added essentially zero *standing* queue. The advance-notice
   timing was directly observable: each invitation arrived 0–20 ms before the
   previous backlog's last byte left the host, i.e., right at the configured
   15 ms.
   -

   165 KB is the "correct" byte threshold for 15 ms at 86 Mb/s. It is the
   best a byte-based tuner could do for the fast path, and it costs ~0.6 s of
   standing queue the moment the same connection runs at 2 Mb/s. Conversely
   the 15 KB value that suits 2 Mb/s cost ~25% of goodput there (sendmsg
   gating, §5) and is far too small for the fast path. The time-based option
   needed no retuning across the 40× rate change — this is the draft's §5
   argument, observed directly.
   -

   Throughput cost of the mechanism: none observed. The REPLENISH_TIME run
   was the fastest run of the set on the unshaped path. (The
   default-configuration baseline at 28.7 Mb/s hit early losses and CUBIC
   recovers slowly at this RTT; the point is that keeping the source buffer
   nearly empty did not prevent full bandwidth utilization, including
   bandwidth *discovery* from slow start — a concern I had about using the
   delivery-rate estimator, which did not materialize.)
   -

   Within a single 23 s connection on the unshaped path, the same 15 ms
   setting tracked the delivery rate from 1.5 Mb/s (early slow start) to ~90
   Mb/s with correct invitations throughout.

*5. Observation: byte-threshold mechanisms that gate write() distort
application behavior*

Linux's TCP_NOTSENT_LOWAT gates sendmsg() as well as poll(): once the
unsent backlog crosses the threshold, a large write returns short (in our
runs, 1 MB writes returned ~58 KB, quantized by GSO skb size). Beyond the
API surprise, this had two measurable effects: the tuned 15 KB threshold
lost ~25% goodput at 2 Mb/s, and — a measurement methodology note for
anyone reproducing this — sub-skb short writes coalesce into shared skbs,
so SO_TIMESTAMPING OPT_ID reports collapse to one per merged skb
(documented Linux behavior), silently thinning per-write latency data. The
draft's notification-only stance avoids both problems — hence suggestion 3
in §1.

*6. Finding: the readiness rule needs a sub-MSS clause, or Nagle — even
with Minshall's modification — traps it at one write per RTT*

The finding, and the fix, up front: a REPLENISH_TIME (or byte-lowat) sender
making sub-MSS-tailed writes with Nagle enabled phase-locks at one write
per RTT. The cure is to treat any sub-MSS unsent backlog as writable while
Nagle is enabled — a one-clause extension of the "empty backlog is always
writable" rule, which the patch now implements, which restored the trapped
workload to full link rate, and which turns out to be exactly what XNU has
long shipped in its TCP_NOTSENT_LOWAT wakeup logic (suggestion 2 in §1,
where the fix's measurements and boundary conditions are also stated). The
rest of this section is the evidence: the trap, why Minshall's modification
does not prevent it, and the macOS comparison.

Measured, on the implementation as first built — before the sub-MSS clause
was added: with 4 KiB writes on the 2 Mb/s / 172 ms path, a REPLENISH_TIME
sender locked at exactly one write per RTT (4096 B / 172 ms ≈ 190 kb/s),
congestion window pinned near its initial value, flow permanently
application-limited. Setting TCP_NODELAY — and changing nothing else — took
the same workload to full link rate with ~24 ms median source delay.

The mechanism of the trap is a three-way interaction:

   1.

   A 4 KiB write is a few full segments plus a sub-MSS tail (on this path,
   MSS 1228: three segments plus 412 B). Nagle — in Linux, Minshall's
   modification of it — holds the tail while a previously sent sub-MSS packet
   remains unacknowledged, so the unsent backlog never reaches zero: it parks
   at the tail size.
   2.

   The writability threshold (rate × REPLENISH_TIME in bytes) sits *below*
   that tail, because the delivery-rate estimate is itself depressed by the
   resulting one-write-per-RTT behavior. The socket is therefore "correctly"
   unwritable by the estimate: positive feedback, and the flow never escapes.
   3.

   The ACK finally releases the tail, the backlog drains, the wakeup fires
   — one invitation per RTT.

One might expect Minshall's modification to prevent this, since it exists
to avoid Nagle pathologies: it permits a sub-MSS packet whenever no
*previous* sub-MSS packet is unacknowledged, i.e., roughly one small packet
per RTT. It does not help here, because the loop consumes exactly that
allowance in phase-lock. The invitation fires at the moment the previous
tail departs — that departure is what makes the backlog reach zero — the
application writes immediately, and the new write's tail therefore always
finds the just-departed small packet still unacknowledged: held one RTT,
released by its ACK, whose processing issues the next invitation. Minshall
prevents small packets from *accumulating*; it cannot help a sender whose
write clock is phase-locked to the small-packet ACK clock. The measured
signature matches this to the byte: exactly one write per RTT, with the
unsent backlog parked at exactly 4096 − 3×1228 = 412 B.

Nor is the trap specific to TCP_REPLENISH_TIME or to my patch: it
reproduces with plain TCP_NOTSENT_LOWAT (whose semantics the patch leaves
unchanged, and which behaves identically on stock Linux) whenever the byte
threshold is below one MSS. Measured: TCP_NOTSENT_LOWAT = 187 with 512 B
writes on the 100 kb/s path collapsed to 15.7 kb/s of ~90 available — each
write held by Minshall–Nagle until the previous write's ACK. A time-based
threshold simply *inherits* this pre-existing lowat behavior whenever its
byte equivalent (rate × REPLENISH_TIME) falls below one MSS — and unlike a
byte threshold, it will cross that line silently as the rate drops, which
is why the guidance of §1 is worth stating.

Platform comparison: the lock does *not* reproduce on macOS. The same
workload (TCP_NOTSENT_LOWAT = 187, 512 B writes, kevent/EVFILT_WRITE, same
100 kb/s path, ~180 ms RTT) sustained ~79 kb/s — effectively full rate —
with sub-MSS data packets pipelined several per RTT. Reading XNU explains
why Darwin is immune, at two independent layers:

   -

   Its kevent readiness check, tcp_notsent_lowat_check(), explicitly
   reports the socket writable whenever the unsent backlog is nonzero but
   smaller than one MSS and Nagle is enabled ("When Nagle's algorithm is not
   disabled, it is better to wakeup the client until there is atleast one
   maxseg of data to write"). The application is therefore invited until a
   full segment's worth pools, which then passes tcp_output's unconditional
   len >= maxseg send test.
   -

   Its Nagle implementation uses a different Minshall variant: a single
   flag (TF_MAXSEGSNT, "the last segment sent was full-sized") rather than
   Linux's "is a previously sent sub-MSS packet still unacknowledged"
   (snd_sml). In the 4 KiB-write pattern the trailing partial always directly
   follows full segments, so Darwin always sends it immediately, while Linux's
   variant always finds the previous small packet unacked and always holds it.

The trap is therefore specific to stacks that combine Linux's form of
Minshall–Nagle with an unsent-byte readiness threshold lacking a sub-MSS
clause — which makes the fix (§1, suggestion 2) less a proposal of mine
than existing Apple practice, independently converged on.

The notification *machinery*, for its part, is blameless: Linux
re-evaluates write readiness on both ACK and transmit events (the latter
since 2022, commit 4bfe744ff164, "tcp: fix potential xmit stalls caused by
TCP_NOTSENT_LOWAT" — the analogous wakeup problem, fixed long ago).
Readiness can only change at those points, since the backlog shrinks only
on transmit and the rate/RTT estimates update only on ACKs, and poll()
additionally re-evaluates synchronously. For the same reason, a timer-based
wakeup would not help: no input to the estimate changes between those
events.

*7. Availability*

Linux patch (new sockopt, ~75 lines against net/ipv4 + net/ipv6) and the
measurement tool (epoll + SO_TIMESTAMPING sender) available on request;
intending to post the patch as an RFC to netdev.


Jonathan Lennox

8x8 / Jitsi

[e-mail]