MIME-Version: 1.0
References: 
 <CAKx+b+YWjWFd61zSbCQjX2zzTfPVo3=8rXS24wQ_4WVhHo9FrA@mail.gmail.com>
In-Reply-To: 
 <CAKx+b+YWjWFd61zSbCQjX2zzTfPVo3=8rXS24wQ_4WVhHo9FrA@mail.gmail.com>
From: Jonathan Lennox <jonathan.lennox42@gmail.com>
Date: Thu, 3 Apr 2025 16:18:37 -0400
Message-ID: 
 <CAKx+b+Z5h4j4veKC5ws+PH5DsM58BGrK0PQuU=5jLDjkzF+zKg@mail.gmail.com>
To: sbm@ietf.org
Content-Type: multipart/alternative; boundary="000000000000440aa90631e57a73"
Message-ID-Hash: MDJS4F33OA6YWW3UERXANOEZVFHOLV3Q
Precedence: list
Subject: =?utf-8?q?=5Bsbm=5D_Re=3A_Kernel_API_invariants_and_surprising_behavior_in_T?=
	=?utf-8?q?CP=5FNOTSENT=5FLOWAT?=
Archived-At: 
 <https://mailarchive.ietf.org/arch/msg/sbm/p4A4p60_bbnwogYeZTXvI2jNETQ>

--000000000000440aa90631e57a73
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I discussed this with some people at the IETF, and it turns out I was wrong
-- this is not actually an invariant that the Linux kernel assumes.

In fact, Linux already has the property that epoll() and select() on a TCP
socket don't return writable until the available space in the socket's send
buffer is at least half the amount of currently queued data, even though
(as far as I can tell from my experiments), a write to the socket will
succeed as soon as any space at all is available (i.e. once data has been
acknowledged).

So, I think a macOS-semantics implementation of TCP_NOTSENT_LOWAT (or a
desired-semantics implementation of TCP_REPLENISH_TIME) would not do the
violence I feared to the Linux kernel's assumptions about sockets, and
hopefully would be relatively straightforward to write.  Instead, it would
more-or-less be modifying the logic that decides the writability decision
above.

Though I still ask if there are any Linux kernel developers here, or if
anyone knows any who can be recruited to help.

On Wed, Mar 12, 2025 at 1:36=E2=80=AFPM Jonathan Lennox <jonathan.lennox42@=
gmail.com>
wrote:

> One surprising thing about the original macOS semantics for
> TCP_NOTSENT_LOWAT (and thus the intended semantics for TCP_REPLENISH_TIME=
)
> is that it breaks an implied kernel invariant.  I hypothesize that this i=
s
> why the Linux implementation behaves differently than the macOS one.
>
> This invariant is that the status returned by select() / kevent() /
> epoll() is *true*: that if those system calls say that a socket isn't
> writable, then it really isn't, and attempting to write to it will block,
> or return EWOULDBLOCK / EAGAIN.
>
> The macOS semantics of TCP_NOTSENT_LOWAT instead break this - select() or
> kevent() will claim that a socket is not writable when it has more than
> TCP_NOTSENT_LOWAT bytes in its buffer, even though a write will in fact
> succeed up to SO_SNDBUF bytes.  Thus, once TCP_NOTSENT_LOWAT is set, ther=
e
> is no way to tell on macOS if a write to the socket will actually succeed
> or block.
>
> I believe this invariant is pretty baked in to the Linux kernel code, and
> that's why the implementation made it so that it was in fact impossible t=
o
> write more than TCP_NOTSENT_LOWAT bytes to the socket.  (I'm not an exper=
t
> in the Linux socket code, but it looks like it uses the same code to
> trigger epoll and to decide when a tcp endmsg() call truncates its write.=
)
>
> I'm thinking that it may be cleaner from an API design perspective
> (though, sadly, less portable) to instead add new kevent and epoll event
> types specifically for when the socket hits its low-water point, whether =
by
> size or by time, and leave the existing writeable events to their origina=
l
> semantic.  (This wouldn't allow triggering low-water from select(), but I
> don't think that's a huge hardship for modern code.)
>
> Are there Linux kernel developers on this list?  I suspect this will need
> input from that community.
>

--000000000000440aa90631e57a73
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I discussed this with some people at the IETF, and it turn=
s out I was wrong -- this is not actually an invariant that the=C2=A0Linux =
kernel assumes.<div><br></div><div>In fact, Linux already has the property =
that epoll() and select() on a TCP socket don&#39;t return writable=C2=A0un=
til the available space in the socket&#39;s send buffer is at least half th=
e amount of currently queued data, even though (as far as I can tell from m=
y experiments), a write to the socket will succeed as soon as any space at =
all is available (i.e. once data has been acknowledged).</div><div><br></di=
v><div>So, I think a macOS-semantics implementation of TCP_NOTSENT_LOWAT (o=
r a desired-semantics implementation of TCP_REPLENISH_TIME) would=C2=A0not =
do the violence I feared to the Linux kernel&#39;s assumptions about socket=
s, and hopefully would be relatively straightforward to write.=C2=A0 Instea=
d, it would more-or-less be modifying the logic that decides the writabilit=
y decision above.</div><div><br></div><div>Though I still ask if there are =
any Linux kernel developers here, or if anyone knows any who can be recruit=
ed to help.</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" clas=
s=3D"gmail_attr">On Wed, Mar 12, 2025 at 1:36=E2=80=AFPM Jonathan Lennox &l=
t;<a href=3D"mailto:jonathan.lennox42@gmail.com" target=3D"_blank">jonathan=
.lennox42@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quot=
e" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)=
;padding-left:1ex"><div dir=3D"ltr">One surprising thing about the original=
 macOS semantics for TCP_NOTSENT_LOWAT (and thus the intended semantics=C2=
=A0for TCP_REPLENISH_TIME) is that it breaks an implied kernel invariant.=
=C2=A0 I hypothesize that this is why the Linux implementation behaves diff=
erently than the macOS one.<div><br></div><div>This invariant is that the s=
tatus returned by select() / kevent() / epoll() is <b>true</b>: that if tho=
se system calls say that a socket isn&#39;t writable, then it really isn=
9;t, and attempting to write to it will block, or return EWOULDBLOCK / EAGA=
IN.</div><div><br></div><div>The macOS semantics of TCP_NOTSENT_LOWAT inste=
ad break this - select() or kevent() will claim that a socket is not writab=
le=C2=A0when it has more than TCP_NOTSENT_LOWAT bytes in its buffer, even t=
hough a write will in fact succeed up to SO_SNDBUF bytes.=C2=A0 Thus, once =
TCP_NOTSENT_LOWAT is set, there is no way to tell on macOS if a write to th=
e socket will actually succeed or block.</div><div><br></div><div>I believe=
 this invariant is pretty baked in to the Linux kernel code, and that&#39;s=
 why the implementation made it so that it was in fact impossible to write =
more than TCP_NOTSENT_LOWAT bytes to the socket.=C2=A0 (I&#39;m not an expe=
rt in the Linux socket code, but it looks like it uses the same code to tri=
gger epoll and to decide when a tcp endmsg() call truncates its write.)</di=
v><div><br></div><div>I&#39;m thinking that it may be cleaner from an API d=
esign perspective (though, sadly, less portable) to instead add new kevent =
and epoll event types specifically for when the socket hits its low-water p=
oint, whether by size or by time, and leave the existing writeable events t=
o their original semantic.=C2=A0 (This wouldn&#39;t allow triggering low-wa=
ter from select(), but I don&#39;t think that&#39;s a huge hardship for mod=
ern code.)</div><div><br></div><div>Are there Linux kernel developers on th=
is list?=C2=A0 I suspect this will need input from that community.</div></d=
iv>
</blockquote></div>

--000000000000440aa90631e57a73--