[sbm] Alternative Linux ways to monitor send buffers

[Jonathan Lennox <[e-mail]>]

In the absence of TCP_REPLENISH_TIME or a "working" TCP_NOTSENT_LOWAT on
Linux, I thought I'd mention some other Linux approaches that could be used
to monitor send buffers.

1.  The linux ioctl SIOCOUTQNSD returns the current amount of unsent data
waiting in a socket, so a socket could be periodically monitored until this
value falls below a desired threshold.  This would likely be a significant
amount of overhead in a production application, however, so probably isn't
a great solution.

2. The Linux SO_TIMESTAMPING sockopt can be used to give an application
callback whenever the data in a write (or send, or sendmsg) call has been
queued for transmission to a network device.  WIth the flags
SOF_TIMESTAMPING_TX_SCHED | SOF_TIMESTAMPING_SOFTWARE |
SOF_TIMESTAMPING_OPT_ID | SOF_TIMESTAMPING_OPT_ID_TCP |
SOF_TIMESTAMPING_OPT_TSONLY, a notification will be available on the
socket's error queue which includes the offset of the data transmitted, so
if the application keeps track of the total amount of data sent, it can
compute the remaining amount of unsent data, with somewhat less overhead
than option 1.  This does, however, have the disadvantage that you get an
extra kernel notification for every write, and the granularity of the
notifications you get depend on the granularity of your write calls, which
depending on your application architecture might be a poor fit to when you
want your notifications; but potentially some sort of double-buffering
approach could be done.

Does anyone have any other suggestions?