Re: [Idr] New Version Notification for draft-spaghetti-idr-bgp-sendholdtimer-05.txt

[Jeffrey Haas <[e-mail]>]

Enke,

On Fri, Aug 19, 2022 at 10:24:54AM -0700, Enke Chen wrote:
> Yes, there needs to be a default value, and it should be configurable.
> That's what we recommend in the draft.

You and Robert are both pointing out that I missed the "or otherwise thirty
minutes after the BGP session is established".  That was my error.

> > Portions of the BGP FSM set timer values depending on what point we are at
> > in the FSM's execution.  It's therefore reasonable to pick a very high value
> > and always start the timer upon session startup and then potentially reduce
> > the timer after we've achieved end-of-rib.  If end-of-rib isn't going to be
> > received, that lower timer threshold may be reasonable to set after some
> > drop-dead time; alternatively just stick with the very high value.
> >
> 
> For such a corner case, IMO we should keep the solution simple.
> It seems that the key is to make the timeout value large (and
> configurable) so we don't generate false positives.

With the "or otherwise" detail I missed, what we'd be looking at according
to your procedures is 30 minutes (no end of rib), set the user timeout, and
minimally 10 minutes or 5*holdtime if larger than 10.  So, stuck state
during initial sync could linger up to 40 minutes or longer.

Is that your intent?

> > Similarly applicable to both drafts, I'm wondering if there may be benefit
> > in signaling the peer what the sendholdtimer (or equivalent tcp-user timer)
> > values are.  It's effectively a warning, and a hint to the remote peer's
> > schedulers to make sure that some amount of data is drained in a timely
> > enough fashion.  The NOTIFICATION subcode we've started discussing for
> > signaling that the sender has torn things down is the other half of this
> > equation.
> 
> The signaling mechanism for TCP user-timeout is already specified in
> RFC5482. It's optional.

I was thinking specifically of a BGP option to provide for implementations
that don't want to regularly be watching for changing TCP values.  For
example, exchanged in a capability.

Reading deeper into RFC 5482, we see some additionally interesting things:

:   In the absence of an application-specified user timeout, the TCP
:   specification [RFC0793] defines a default user timeout of 5 minutes.

... with several refinements.

This section apparently is arguing that if we're implementing base
conformant TCP that we shouldn't be getting into this problem in the first
place at the TCP layer.

Section 4.2 discussing TCP Keep-alives notes that some stacks may have
different interactions between keepalives vs. user timeout.

Time to go read some TCP code...


> > With regard to the TCP specific feature, I still have personal unease with
> > what the timer implies.  The general desire articulated by many in these
> > threads is "the peer makes progress and doesn't stay blocked forever".
> > Certainly the TCP user timeout would accomplish that.  However, the timer is
> > versus "unacknowledged data".  What this means is not "we've made no
> > progress", rather "stuff was enqueued X seconds ago and unacknowledged".
> >
> 
> The TCP user-timer applies to "un-delivered data", including both
> "unacked data" and "buffered but untransmitted data".

Understood.  Again, my unease is that the timer says "something enqueued
wasn't delivered in X time".  Do we want that, or do we simply want "this
TCP session continues to make progress?"

I suspect that answer might depend on how long the timers are.

> > I think the above illustrates a key detail the Working Group will need
> > consensus on: Do we care if we're not making progress (perhaps at all), or
> > are we willing to put a timeliness requirement on the contents?
> >
> 
> BGP only knows when the local TCP write-buffer is full, and that is
> only one specific case. There are several other cases where the TCP
> write-buffer is not full, but can also result in stale routes:
> 
>     - BGP messages are not delivered for a long time, either due to
> zero-window or lost ack.

Indeed.  And the BGP implementation is generally ignorant of what has been
enqueued to TCP vs. delivered.  So, 10 minutes of nothing but keepalives may
not be the reason to shut down the session, but 10 minutes of updates not
being delivered may be important.  Or even sooner for undelievered route
refresh messages.

> It can take a long time for the TCP buffer to be filled up when
> routing is relatively stable.
> 
> That's why I believe a solution at the BGP level would be incomplete
> and perhaps even more complex. On the other hand, TCP user-timeout is
> specifically designed to handle these cases. It's more complete and
> simpler.

As long as you're fine with the one-bit signal of "user timeout has gone
down".  That's not necessarily a clear win for some of the people who've
responded to the thread.

Perhaps zero surprise, experience in BFD has provided similar "sometimes one
bit of signaling isn't enough".  Thus, the WG needs further discussion.

-- Jeff