Re: [Idr] Fwd: New Version Notification for draft-spaghetti-idr-bgp-sendholdtimer-05.txt

[Jeffrey Haas <[e-mail]>]

Claudio,

On Thu, Aug 04, 2022 at 09:39:09AM +0200, Claudio Jeker wrote:
> On Wed, Aug 03, 2022 at 01:03:17PM -0400, Jeffrey Haas wrote:
> > The underlying conditions we want from my perspective are:
> > - We have detected we are unable to write further data to the TCP connection
> >   and have more data to send.  Certainly EWOULDBLOCK would be a common
> >   example.  How about EAGAIN?  Depends on the stack.  Other conditions such as
> >   ENOBUFS, EINTR, ENOMEM, ENOSPC lead to questions as to whether this is a
> >   useful POSIX indication that you can't write vs. you can't write because TCP
> >   is not progressing.
> 
> EWOULDBLOCK is normaly the same as EAGAIN (at least on anything modern).
> EINTR should be handled anyway because it is caused by signal deliver and
> has nothing todo with this issue. Generally after EINTR the same call is
> retried immediatly.
> 
> ENOBUFS, ENOMEM and ENOSPC are proper error conditions that need some
> action (normally a fatal write error that causes the connection to close).
> Also ENOMEM is not an error send(2) should return according to POSIX.

My faith that a particular implementation follows POSIX or not, is low at
the best of times.  I've been doing this stuff for Too Long.

I, and some of the other vendors on this list, also have the joy that our
TCP stacks have been doctored for hardware integration and non-stop routing
purposes.  This makes the situation even more murky.

Enke has done us a nice service in pointing out the Linux TCP_USER_TIMEOUT
feature which nicely encapsulates much of the behavior we want, but that
won't be universally supported either.  And even in that case, we likely
want to know at the application layer that was the reason the session closed
so we can locally take action on that.

My caveat here is that it might seem "obvious" what to write in terms of
observable socket behavior... but it's not, and trying to discuss too much
at that layer will likely not be as successful as we like.

> > - That said, if the TCP session starts draining and data starts moving, and
> >   we become able to write, the timer should be canceled.  
> >  
> >   + How does that manifest vs. kevent/select/poll/etc.?
>  
> In OpenBGPD a timeout is used that is rearmed whenever data is enqueued.
> If the timer fires, the socket did not become writeable for sendholdtimer
> interval and the session is reset. This works with poll, select, kevent, etc.

Offering observations from my own implementation, just because your timer
fired didn't mean the sockets didn't make progress in draining.  

To work around that condition, we peek at the TCP state when the holdtimer
fires. 

> > There are a number of other messier edge cases where you're not at a zero
> > window, but still can't write to the socket.  Basically any case where you
> > can't write pending data to the socket for the sendholdtimer interval is
> > probably a Good Enough condition.
> 
> IMO if a BGP speaker was not able to send out (or enqueue to send out) a
> message for sendholdtimer interval then it is clear that the BGP session
> is not functioning and the other side is lying about its holdtimer
> (because we know there was no keepalive sent for more than the holdtimer
> time).

We agree here.

The implication is to be careful in the document about using a zero window
as the gating condition for the feature.

-- Jeff