[sbm] Kernel API invariants and surprising behavior in TCP_NOTSENT_LOWAT

[Jonathan Lennox <[e-mail]>]

One surprising thing about the original macOS semantics for
TCP_NOTSENT_LOWAT (and thus the intended semantics for TCP_REPLENISH_TIME)
is that it breaks an implied kernel invariant.  I hypothesize that this is
why the Linux implementation behaves differently than the macOS one.

This invariant is that the status returned by select() / kevent() / epoll()
is *true*: that if those system calls say that a socket isn't writable,
then it really isn't, and attempting to write to it will block, or return
EWOULDBLOCK / EAGAIN.

The macOS semantics of TCP_NOTSENT_LOWAT instead break this - select() or
kevent() will claim that a socket is not writable when it has more than
TCP_NOTSENT_LOWAT bytes in its buffer, even though a write will in fact
succeed up to SO_SNDBUF bytes.  Thus, once TCP_NOTSENT_LOWAT is set, there
is no way to tell on macOS if a write to the socket will actually succeed
or block.

I believe this invariant is pretty baked in to the Linux kernel code, and
that's why the implementation made it so that it was in fact impossible to
write more than TCP_NOTSENT_LOWAT bytes to the socket.  (I'm not an expert
in the Linux socket code, but it looks like it uses the same code to
trigger epoll and to decide when a tcp endmsg() call truncates its write.)

I'm thinking that it may be cleaner from an API design perspective (though,
sadly, less portable) to instead add new kevent and epoll event types
specifically for when the socket hits its low-water point, whether by size
or by time, and leave the existing writeable events to their original
semantic.  (This wouldn't allow triggering low-water from select(), but I
don't think that's a huge hardship for modern code.)

Are there Linux kernel developers on this list?  I suspect this will need
input from that community.