[sbm] Re: Kernel API invariants and surprising behavior in TCP_NOTSENT_LOWAT

[Jonathan Lennox <[e-mail]>]

I discussed this with some people at the IETF, and it turns out I was wrong
-- this is not actually an invariant that the Linux kernel assumes.

In fact, Linux already has the property that epoll() and select() on a TCP
socket don't return writable until the available space in the socket's send
buffer is at least half the amount of currently queued data, even though
(as far as I can tell from my experiments), a write to the socket will
succeed as soon as any space at all is available (i.e. once data has been
acknowledged).

So, I think a macOS-semantics implementation of TCP_NOTSENT_LOWAT (or a
desired-semantics implementation of TCP_REPLENISH_TIME) would not do the
violence I feared to the Linux kernel's assumptions about sockets, and
hopefully would be relatively straightforward to write.  Instead, it would
more-or-less be modifying the logic that decides the writability decision
above.

Though I still ask if there are any Linux kernel developers here, or if
anyone knows any who can be recruited to help.

On Wed, Mar 12, 2025 at 1:36 PM Jonathan Lennox <[e-mail]>
wrote:

> One surprising thing about the original macOS semantics for
> TCP_NOTSENT_LOWAT (and thus the intended semantics for TCP_REPLENISH_TIME)
> is that it breaks an implied kernel invariant.  I hypothesize that this is
> why the Linux implementation behaves differently than the macOS one.
>
> This invariant is that the status returned by select() / kevent() /
> epoll() is *true*: that if those system calls say that a socket isn't
> writable, then it really isn't, and attempting to write to it will block,
> or return EWOULDBLOCK / EAGAIN.
>
> The macOS semantics of TCP_NOTSENT_LOWAT instead break this - select() or
> kevent() will claim that a socket is not writable when it has more than
> TCP_NOTSENT_LOWAT bytes in its buffer, even though a write will in fact
> succeed up to SO_SNDBUF bytes.  Thus, once TCP_NOTSENT_LOWAT is set, there
> is no way to tell on macOS if a write to the socket will actually succeed
> or block.
>
> I believe this invariant is pretty baked in to the Linux kernel code, and
> that's why the implementation made it so that it was in fact impossible to
> write more than TCP_NOTSENT_LOWAT bytes to the socket.  (I'm not an expert
> in the Linux socket code, but it looks like it uses the same code to
> trigger epoll and to decide when a tcp endmsg() call truncates its write.)
>
> I'm thinking that it may be cleaner from an API design perspective
> (though, sadly, less portable) to instead add new kevent and epoll event
> types specifically for when the socket hits its low-water point, whether by
> size or by time, and leave the existing writeable events to their original
> semantic.  (This wouldn't allow triggering low-water from select(), but I
> don't think that's a huge hardship for modern code.)
>
> Are there Linux kernel developers on this list?  I suspect this will need
> input from that community.
>