Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

["Jakob Heitz (jheitz)" <[e-mail]>]

If you tell the socket to shutdown and then close, it will attempt to
send everything in the queue with the FIN at the end.
Then wait for the FIN ACK and all manner of nonsense to bore rtr-B to tears.
So, to get on with it, send the RST.

Next question is what to do if GR is in effect.
rtr-A will dutifully retain all the routes from rtr-B and Job's beloved WITHDRAW
will still not happen.
The new session will come up (maybe), rtr-B will send all its routes again and
(if it doesn't get stuck again) will send its EOR. Only now can Job breathe easy.

Might we need a new bit in the GR capability in the OPEN message?
"WITHDRAW ALL MY ROUTES NOW"

Regards,
Jakob.

From: Idr <[e-mail]> On Behalf Of Robert Raszuk
Sent: Tuesday, December 15, 2020 3:19 PM
To: Job Snijders <[e-mail]>
Cc: [e-mail]
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

Hi Job,

Putting all other concerns aside I have few questions ...

1. Is this BGP which should trigger the session RST or FIN or TCP ?

2. If this is BGP (TCP would not be aware of HOLD_SEND) how exactly do we know that peer's window is 0 for HOLD_SEND TIME ?

3. Which TCP socket option will return BGP an error that for the duration of X sec window for a given peer was 0 ? I presumed even if it jumped for 100 ms above 0 the timer would be reset indicating peer is still alive ?

From your bgpd example you are not checking anything other then BGP's ability to write to out queue. So is this the suggestion now forgetting all about TCP layer ? Simply if I can not write anything to a peer for over X sec RST the session ?

Hi John,

I think the suggestion is to add a second HOLD_SEND TIME different from normal HOLD TIME.

Also there could be lost of different type of peers so unless HOLD_SEND would be say 5 x HOLD putting all peers under same time value may be suboptimal.

Thx,
R.


On Tue, Dec 15, 2020 at 10:54 PM Job Snijders <[e-mail]<mailto:[e-mail]>> wrote:
On Tue, Dec 15, 2020 at 09:57:47PM +0100, Christoph Loibl wrote:
> Thanks for answering my question in more detail. Maybe I was unclear
> (but reading your email I think we are talking about the same).
> > On 15.12.2020, at 21:00, John Scudder <[e-mail]<mailto:[e-mail]>> wrote:
> >
> > I think you are talking about this scenario. I’ll copy the example
> > from Rob’s message cited above:
> >
> >   rtr-A                   rtr-B
> >   (congested c-p)         (uncongested c-p)
> >   send window: >0         send window: 0
> >   recv window: 0          recv window: >0
> >
> > In this case we expect:
> >  a) rtr-B does not send any BGP packet (KEEPALIVE/UPDATE/NOTIFICATION)
> > to rtr-A in normal operating circumstances.
> >  b) rtr-A does not expect any KEEPALIVE/UPDATE packets from rtr-B. The
> > session remains established even if no packet is received in the
> > holdtime.
> >  c) rtr-A continues to send KEEPALIVE packets to rtr-B.
>
> The part I have a problem to understand is b). It is clear that rtr-A
> will not receive any packets from rtr-B because rtr-B cannot send them
> (send window: 0). But does "rtr-A does not expect any KEEPALIVE/UPDATE
> packets from rtr-B” mean that rtr-A has essentially suspended its
> hold-timer until it is ready to receive new messages and opens up its
> recv window? If yes, why? I would expect timers to run independently
> of the transport protocol.

Yeah, I'd expect that too. We've seen congested BGP implementations
continue to send KEEPALIVEs but not accept (or send!) other BGP
messages. And rtr-B's attempts at KEEPALIVE just be TCP ACked with zero
window.

I'd argue in the above scenario rtr-A is simply broken and rtr-B MUST
proceed to close down the session towards rtr-A, rtr-B must cleanup and
generate WITHDRAWs for any routes pointing to rtr-A. By doing the
clean-up rtr-B does both itself and rtr-A a favor. If the issue was
transcient rtr-A and rtr-B will re-establish a few minutes later
(IdleHoldTimer, right?) and things will normalize.

Arguably and measurably, rtr-A is operating its Loc-RIB (forwarding)
based on stale routing information (assuming rtr-A is working at all!):
rtr-A has not received any WITHDRAWs, UPDATEs (or somewhat less
importantly KEEPALIVEs) from rtr-B.

Rtr-B is fully aware of this stale situation, because rtr-B was not able
to write these BGP messages to the network: the messages are still in
OutQ. Rtr-A didn't accept any KEEPALIVE (or UPDATE/WITHDRAW) from
rtr-B.

How to solve this? Claudio Jeker took a look at what it would take in
OpenBGPD and came up with the (tiny!) following patch, should be
readable to most: https://marc.info/?l=openbsd-tech&m=160796802508185&w=2

Ben Cox helped me create a 'EBGP peer from hell': a publicly accessible
EBGP multihop instance which can reliably produce the undesirable
TCP/BGP behavior we're discussing here. This 'peer from hell' will do
the OPEN exchange but then manipulates the TCP recvwindow towards zero.

All BGP implementations tested so far (5 famous ones) appear vulnerable
because they continue to consider the BGP session healthy & stable
(meanwhile OutQ keeps growing endlessly and zero BGP messages go across
the wire).

One network operator (with thousands of EBGP sessions in the DFZ)
reported to me the above stalled-TCP scenario is *not* a common case on
the Internet. On a normal day, a network operator will see no (zero)
sessions stuck this way, which leads me to believe 'recvwind=0' ...
*for the duration of the hold timer* is a very strong indicator for a
really broken situation which should be attempted to automatically
resolve.

I believe BGP implementations are not helping any known deployment
scenarios by *not* disconnecting a stuck peer, however on the other we
now know about various operational examples where honoring recvwind=0
for (hours, days) longer than $holdtimer led to global scale problems.

As the 'not-at-all progressing OutQ' situation seems somewhat rare in
the wild (yet continues to happen from time to time) I think it is worth
discussing & documenting how implementers can attempt to avoid this
state from happening. It might help make the Internet 1% more robust.

BGP implementers (or operators wanting to test their equipment) feel
free to contact me off-list if you'd like to set up an EBGP multihop
session towards the 'peer from hell' testbed. Testing potential
solutions this way is quite easy, the behavior can be triggered within a
few seconds.

Kind regards,

Job

ps. At this moment we have (1) an attempt at problem description, (2) a
demonstration BGP-4 implementation of a 'problem causer', and (3) a
different BGP-4 implementation with a 'solution'. This enables IDR to
test interopability & (potentially revised) protocol compliance,
hopefully moving the problem a bit from theoretical to practical
reality? :)