[Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

[Job Snijders <[e-mail]>]

Dear group,

Not too long ago an incident [1] in one Autonomous System resulted in
the global Internet being unusable in many parts of the world for
multiple hours. Some have reported the root cause was a 'configuration
error', however I believe much of the observed communication blackouts
in the global routing system stemmed from a pre-existing condition: a
specific implementation property present in multiple implementations
currently in use in the default-free zone. 

Usually when an incident happens in one AS, affected parties can through
unilateral action 'route around the problem', but the ability to 'route
around problems' critically depends on the ability to distribute
WITHDRAW or UPDATE messages. When messages are not processed, what
generally was assumed to be a unilaterally solvable problem, now requires
coordination between *all* neighbors of the suffering AS.

The global routing system requires every participant to process BGP
messages, because the alternative is intervention on thousands of BGP
devices to manually shutdown thousands of BGP sessions disconnecting the
AS suffering from an incident, to help the rest of the default-free
zone. I speak from experience when saying that coordinating a disconnection
of an AS at global scale is incredibly hard and slow, any many approval
levels must be worked through. It takes *hours* of phone calls & email
chains, a time window during which internet traffic is routed towards
stale (now blackholing) locations.

In the average ISP's network design using IBGP Route Reflectors, these
blackout effects are aggravated when BGP sessions landing in such
devices are not terminated when TCP causes the BGP session to stall.

The problem of how TCP and BGP-4 can interact has been discussed before,
but I'm not sure the working group followed up with any publication
detailing the problem and the solution.

    https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/

Does everyone agree BGP-4 sessions MUST be terminated using a TCP RST
(instead of a BGP-4 Cease NOTIFICATION) if the peer has indicated for
the duration of the Hold Timer that the TCP receive window is zero?
I'm fine with there being buttons to make this different, but the
default for routers in the global Internet routing system should be to
consider the remote peer to be 'a lost cause' when it won't accept new
BGP messages for the duration of the hold timer.

Perhaps RFC 4271 Section 6.5 should be amended as following:

OLD:
    If a system does not receive successive KEEPALIVE, UPDATE, and/or
    NOTIFICATION messages within the period specified in the Hold Time
    field of the OPEN message, then the NOTIFICATION message with the
    Hold Timer Expired Error Code is sent and the BGP connection is
    closed.

NEW:
    If a system does not receive (or is unable to send) successive
    KEEPALIVE, UPDATE, and/or NOTIFICATION messages within the period
    specified in the Hold Time field of the OPEN message, then the
    NOTIFICATION message with the Hold Timer Expired Error Code is sent
    and the BGP connection is closed. If the NOTIFICATION message cannot
    be send the BGP connection is closed.

This is an ongoing problem. I suspect the BGP Nyancat's discoloration at
the left most eye might have been caused by an active TCP session
keeping a stale BGP session alive. But also the observations from "BGP
Zombies: an Analysis of Beacons Stuck Routes" [3] could be explained by
the problematic interaction between TCP and BGP.

I appreciate the work the IDR working group has done to *SOFTEN* the
blow from implementation defects on global routing (RFC 7606 is a
brilliant example of this), but I fear in this case there is no subtle
way to say goodbye when the peer doesn't process messages in a timely
fashion. It might be good to document this.

Kind regards,

Job

[1]: https://www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C
[2]: https://labs.ripe.net/Members/cteusche/bgp-meets-cat
[3]: https://www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf