Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

[Gyan Mishra <[e-mail]>]

Hi Job

I had a question about the adj-RIB-in and adj-RIB-out during the TCP paused
BGP stuck state.

Did the rtr-A rib in and rtr-B rib out did that look normal showing 800k in
out ?

So from a NOC perspective the peering looked almost normal except for the
rtr-A red flag with 0 or few routes.

rtr-A GR enabled ?
adj-RIB-in  ? did this show few routes or 0 did it change throughout the
stuck duration
adj-RIab-out 800k routes - Looked Normal

rtr-B GR enabled ?
adj-RIB-in  800k routes - Looked Normal
adj-RIab-out  did this show 800k routes out to the peer mentioned

Also from an interface level for the peer I am guessing was showing 0
traffic in either direction during the 46 minutes you probably also had a
ton of drops on the inter ISP peer in either direction.

That was probably an immediate red flag for the NOC of the stuck state.

So when the NOC discovered the issue they saw the 1 way control plane.

Did the NOC think of bouncing the peer to recover?

Bounce immediately would have forced the withdrawal and immediate
convergence.

Very interesting real world problem and thank you again for bringing to
IDR.  We definitely need to find a solid solution which looks like we are
on our way.


Kind Regards

Gyan

On Wed, Dec 16, 2020 at 11:30 AM Job Snijders <[e-mail]> wrote:

> Dear all,
>
> A follow-up with some PCAP data to illustate the issue
>
> On Tue, Dec 15, 2020 at 08:00:06PM +0000, John Scudder wrote:
> >   rtr-A                   rtr-B
> >   (congested c-p)         (uncongested c-p)
> >   send window: >0         send window: 0
> >   recv window: 0          recv window: >0
> >
> > In this case we expect:
> >  a) rtr-B does not send any BGP packet (KEEPALIVE/UPDATE/NOTIFICATION)
> > to rtr-A in normal operating circumstances.
> >  b) rtr-A does not expect any KEEPALIVE/UPDATE packets from rtr-B. The
> > session remains established even if no packet is received in the
> > holdtime.
> >  c) rtr-A continues to send KEEPALIVE packets to rtr-B.
>
> A PCAP showing the above scenario is available here:
>
>     webhosted decoded: https://www.cloudshark.org/captures/6b120e111c76
>     original .pcap file:
> http://sobornost.net/~job/bgp_tcp_bad_interaction.pcap
>
> rtr-A == 165.254.255.17, rtr-B == 45.138.228.4
> The PCAP was captured on a wiretap applied between these BGP nodes.
>
> rtr-B is trying to send a full routing table (~ 800,000 routes) to
> rtr-A, rtr-A in turn has only a few (stale) downstream routes.
>
> Things are normal during the first 22 seconds (packets 1-149). At packet
> 150 it becomes clear rtr-B is not able to send UPDATEs (or KEEPALIVEs or
> WITHDRAWs) to rtr-A. This situation persisted for ~ 46 minutes, at which
> point I manually killed the sesssion.
>
> During those 46 minutes (wall clock) the OutQ on rtr-B climbed to
> thousands. Even if rtr-B has decent OutQ deduplication (which could
> somewhat hide the detrimental effects of this situation), it is clear
> from the packet capture that rtr-B is not able to follow the spirit and
> intent of the BGP Hold Timers: BGP communication is completely and fully
> stalled in one direction. All of rtr-A's BGP messages are TCP ACKed, but
> zero progress is made sending anything from rtr-B to rtr-A.
>
> About the OpenBGPD example solution: OpenBGPD ('bgpd') is an integral
> part of the OpenBSD Network Operating System. The OpenBSD developers are
> responsible for bgpd, userland, the kernel, ssh, TCP subsystem, NIC
> drivers, all of it. Conceptually such a 'complete fullstack
> implementation' is no different than its more impressive siblings Junos
> (a complete operating system with an embedded BGP implementation), IOS
> XR, or SR-OS. Customers are interested in the complete product.
>
> I think vendors are expected to have full ownership of their entire
> product: customers most likely won't care how "Inside the router
> chassis" the BGP daemon(s), kernel(s), hypervisor, filesystem, NIC
> drivers, etc are separate components. What matters is what actually
> happens on the wire between between individual BGP nodes.
>
> I think Takt offered a valuable insight:
>
>     "Usually there is some tx buffering going on. Just because write()
>     was successful doesn't mean a message actually arrived on the other
>     hand. But if write() blocked for an Holdtimer interval it is sure
>     there is an issue."
>     source: https://twitter.com/taktv6/status/1338223595487719436
>
> It's up to each implementation/vendor how to pull up state from lower
> layers into their BGP engine. In the case of OpenBGPD we are fortunate
> to have some infrastructure in place to accomodate this type of
> improvement. Other implementations might have to come up with different
> solutions depending how they designed to handle IO or buffer BGP
> messages.
>
> Compliance testing for the yet-to-be-submitted internet-draft will be
> gauged simply by looking at what happens on the wire between two nodes.
>
> Kind regards,
>
> Job
>
> _______________________________________________
> Idr mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/idr
>
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *



*M 301 502-134713101 Columbia Pike *Silver Spring, MD