Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

[Job Snijders <[e-mail]>]

On Tue, Dec 15, 2020 at 09:57:47PM +0100, Christoph Loibl wrote:
> Thanks for answering my question in more detail. Maybe I was unclear
> (but reading your email I think we are talking about the same).
> > On 15.12.2020, at 21:00, John Scudder <[e-mail]> wrote:
> > 
> > I think you are talking about this scenario. I’ll copy the example
> > from Rob’s message cited above:
> > 
> >   rtr-A                   rtr-B
> >   (congested c-p)         (uncongested c-p)
> >   send window: >0         send window: 0
> >   recv window: 0          recv window: >0	
> > 
> > In this case we expect:
> >  a) rtr-B does not send any BGP packet (KEEPALIVE/UPDATE/NOTIFICATION)
> > to rtr-A in normal operating circumstances.
> >  b) rtr-A does not expect any KEEPALIVE/UPDATE packets from rtr-B. The
> > session remains established even if no packet is received in the
> > holdtime.
> >  c) rtr-A continues to send KEEPALIVE packets to rtr-B.
> 
> The part I have a problem to understand is b). It is clear that rtr-A
> will not receive any packets from rtr-B because rtr-B cannot send them
> (send window: 0). But does "rtr-A does not expect any KEEPALIVE/UPDATE
> packets from rtr-B” mean that rtr-A has essentially suspended its
> hold-timer until it is ready to receive new messages and opens up its
> recv window? If yes, why? I would expect timers to run independently
> of the transport protocol.

Yeah, I'd expect that too. We've seen congested BGP implementations
continue to send KEEPALIVEs but not accept (or send!) other BGP
messages. And rtr-B's attempts at KEEPALIVE just be TCP ACked with zero
window.

I'd argue in the above scenario rtr-A is simply broken and rtr-B MUST
proceed to close down the session towards rtr-A, rtr-B must cleanup and
generate WITHDRAWs for any routes pointing to rtr-A. By doing the
clean-up rtr-B does both itself and rtr-A a favor. If the issue was
transcient rtr-A and rtr-B will re-establish a few minutes later
(IdleHoldTimer, right?) and things will normalize.

Arguably and measurably, rtr-A is operating its Loc-RIB (forwarding)
based on stale routing information (assuming rtr-A is working at all!):
rtr-A has not received any WITHDRAWs, UPDATEs (or somewhat less
importantly KEEPALIVEs) from rtr-B.

Rtr-B is fully aware of this stale situation, because rtr-B was not able
to write these BGP messages to the network: the messages are still in
OutQ. Rtr-A didn't accept any KEEPALIVE (or UPDATE/WITHDRAW) from
rtr-B.

How to solve this? Claudio Jeker took a look at what it would take in
OpenBGPD and came up with the (tiny!) following patch, should be
readable to most: https://marc.info/?l=openbsd-tech&m=160796802508185&w=2

Ben Cox helped me create a 'EBGP peer from hell': a publicly accessible
EBGP multihop instance which can reliably produce the undesirable
TCP/BGP behavior we're discussing here. This 'peer from hell' will do
the OPEN exchange but then manipulates the TCP recvwindow towards zero.

All BGP implementations tested so far (5 famous ones) appear vulnerable
because they continue to consider the BGP session healthy & stable
(meanwhile OutQ keeps growing endlessly and zero BGP messages go across
the wire).

One network operator (with thousands of EBGP sessions in the DFZ)
reported to me the above stalled-TCP scenario is *not* a common case on
the Internet. On a normal day, a network operator will see no (zero)
sessions stuck this way, which leads me to believe 'recvwind=0' ...
*for the duration of the hold timer* is a very strong indicator for a
really broken situation which should be attempted to automatically
resolve.

I believe BGP implementations are not helping any known deployment
scenarios by *not* disconnecting a stuck peer, however on the other we
now know about various operational examples where honoring recvwind=0
for (hours, days) longer than $holdtimer led to global scale problems.

As the 'not-at-all progressing OutQ' situation seems somewhat rare in
the wild (yet continues to happen from time to time) I think it is worth
discussing & documenting how implementers can attempt to avoid this
state from happening. It might help make the Internet 1% more robust.

BGP implementers (or operators wanting to test their equipment) feel
free to contact me off-list if you'd like to set up an EBGP multihop
session towards the 'peer from hell' testbed. Testing potential
solutions this way is quite easy, the behavior can be triggered within a
few seconds.

Kind regards,

Job

ps. At this moment we have (1) an attempt at problem description, (2) a
demonstration BGP-4 implementation of a 'problem causer', and (3) a
different BGP-4 implementation with a 'solution'. This enables IDR to
test interopability & (potentially revised) protocol compliance,
hopefully moving the problem a bit from theoretical to practical
reality? :)