Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

[John Scudder <[e-mail]>]

[all hats on]

Hi Job,

Thanks for bringing this up.

To take the liberty of summarizing your wall of text :-) you’re saying that you believe BGP should tear down its session if it’s unable to send a message for the duration of the hold time. 

Given that the conversation last time was inconclusive I think this is a good thing for the WG to discuss again. If you want to, you (or someone) could turn the idea into a short draft that updates RFC 4271, and we could have a WG adoption discussion about it. It might help focus the discussion but it’s not mandatory.

I’ll point out a few things to start with —

- Making it mandatory to apply hold time to the sending of messages would potentially make BGP peerings less stable. It clearly can’t make them *more* stable. Of course one can argue that if you haven’t been able to send a message for the hold time, the session has failed its metric of usefulness anyway, so any veneer of stability at this point is a harmful sham.
- If I recall correctly, RST doesn’t work (or may not work) if you’re using the MD5 TCP option. Nothing much to be done, but be aware.
- There is nothing stopping an implementation from doing what you describe now. The formalism that keeps you within the letter of 4271 would be that the implementation supplies a configuration option, that you set to enable the behavior. Once you’ve done that, when the implementation notices that the hold time has been exceeded in the outbound direction, it generates a ManualStop event for the session. 

Thanks,

—John

> On Dec 11, 2020, at 2:23 PM, Job Snijders <[e-mail]> wrote:
> 
> 
> Dear group,
> 
> Not too long ago an incident [1] in one Autonomous System resulted in
> the global Internet being unusable in many parts of the world for
> multiple hours. Some have reported the root cause was a 'configuration
> error', however I believe much of the observed communication blackouts
> in the global routing system stemmed from a pre-existing condition: a
> specific implementation property present in multiple implementations
> currently in use in the default-free zone.
> 
> Usually when an incident happens in one AS, affected parties can through
> unilateral action 'route around the problem', but the ability to 'route
> around problems' critically depends on the ability to distribute
> WITHDRAW or UPDATE messages. When messages are not processed, what
> generally was assumed to be a unilaterally solvable problem, now requires
> coordination between *all* neighbors of the suffering AS.
> 
> The global routing system requires every participant to process BGP
> messages, because the alternative is intervention on thousands of BGP
> devices to manually shutdown thousands of BGP sessions disconnecting the
> AS suffering from an incident, to help the rest of the default-free
> zone. I speak from experience when saying that coordinating a disconnection
> of an AS at global scale is incredibly hard and slow, any many approval
> levels must be worked through. It takes *hours* of phone calls & email
> chains, a time window during which internet traffic is routed towards
> stale (now blackholing) locations.
> 
> In the average ISP's network design using IBGP Route Reflectors, these
> blackout effects are aggravated when BGP sessions landing in such
> devices are not terminated when TCP causes the BGP session to stall.
> 
> The problem of how TCP and BGP-4 can interact has been discussed before,
> but I'm not sure the working group followed up with any publication
> detailing the problem and the solution.
> 
>    https://urldefense.com/v3/__https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkPhCc8cBA$
> 
> Does everyone agree BGP-4 sessions MUST be terminated using a TCP RST
> (instead of a BGP-4 Cease NOTIFICATION) if the peer has indicated for
> the duration of the Hold Timer that the TCP receive window is zero?
> I'm fine with there being buttons to make this different, but the
> default for routers in the global Internet routing system should be to
> consider the remote peer to be 'a lost cause' when it won't accept new
> BGP messages for the duration of the hold timer.
> 
> Perhaps RFC 4271 Section 6.5 should be amended as following:
> 
> OLD:
>    If a system does not receive successive KEEPALIVE, UPDATE, and/or
>    NOTIFICATION messages within the period specified in the Hold Time
>    field of the OPEN message, then the NOTIFICATION message with the
>    Hold Timer Expired Error Code is sent and the BGP connection is
>    closed.
> 
> NEW:
>    If a system does not receive (or is unable to send) successive
>    KEEPALIVE, UPDATE, and/or NOTIFICATION messages within the period
>    specified in the Hold Time field of the OPEN message, then the
>    NOTIFICATION message with the Hold Timer Expired Error Code is sent
>    and the BGP connection is closed. If the NOTIFICATION message cannot
>    be send the BGP connection is closed.
> 
> This is an ongoing problem. I suspect the BGP Nyancat's discoloration at
> the left most eye might have been caused by an active TCP session
> keeping a stale BGP session alive. But also the observations from "BGP
> Zombies: an Analysis of Beacons Stuck Routes" [3] could be explained by
> the problematic interaction between TCP and BGP.
> 
> I appreciate the work the IDR working group has done to *SOFTEN* the
> blow from implementation defects on global routing (RFC 7606 is a
> brilliant example of this), but I fear in this case there is no subtle
> way to say goodbye when the peer doesn't process messages in a timely
> fashion. It might be good to document this.
> 
> Kind regards,
> 
> Job
> 
> [1]: https://urldefense.com/v3/__https://www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMkF2w4cg$
> [2]: https://urldefense.com/v3/__https://labs.ripe.net/Members/cteusche/bgp-meets-cat__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMry7Ktyw$
> [3]: https://urldefense.com/v3/__https://www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkO8A78j8Q$
> 
> _______________________________________________
> Idr mailing list
> [e-mail]
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMMXdwc-g$