Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

["Jakob Heitz (jheitz)" <[e-mail]>]

Good point Keyur.
A receiver may be overwhelmed for a long time and not open its TCP window to avoid
silly window syndrome or some other reason. The receiver may still be functional
and able to clear its backlog, albeit in a long time. Resetting such a session
will only make the situation worse. Telling the difference between this case
and a receiver stuck in a bug is difficult.

Regards,
Jakob.

-----Original Message-----
From: Idr <[e-mail]> On Behalf Of Keyur Patel
Sent: Friday, December 11, 2020 5:31 PM
To: Jeff Tantsura <[e-mail]>
Cc: John Scudder <jgs=[e-mail]>; [e-mail]
Subject: Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

One more comment: 

The flapping of the session may result into persistent flaps (more network instability) if the session is brought up prematurely. Most issues are a side effect of excessive throttling of resources at the protocol/system level or bugs in the implementation. That means a manual intervention would be needed before the session is restarted.  

Regards,
Keyur

On 12/11/20, 3:57 PM, "Jeff Tantsura" <[e-mail]> wrote:

    The trade-off is (as often happens) between stability and convergence.
    Given severity, I’d prefer formalized approach rather than implementation artifact ( at mercy of Product Manager in charge ;-))

    Regards,
    Jeff

    > On Dec 11, 2020, at 15:30, Keyur Patel <[e-mail]> wrote:
    > 
    > One comment inlined #Keyur
    > 
    > On 12/11/20, 12:04 PM, "Idr on behalf of John Scudder" <[e-mail] on behalf of jgs=[e-mail]> wrote:
    > 
    >    [all hats on]
    > 
    >    Hi Job,
    > 
    >    Thanks for bringing this up.
    > 
    >    To take the liberty of summarizing your wall of text :-) you’re saying that you believe BGP should tear down its session if it’s unable to send a message for the duration of the hold time. 
    > 
    >    Given that the conversation last time was inconclusive I think this is a good thing for the WG to discuss again. If you want to, you (or someone) could turn the idea into a short draft that updates RFC 4271, and we could have a WG adoption discussion about it. It might help focus the discussion but it’s not mandatory.
    > 
    >    I’ll point out a few things to start with —
    > 
    >    - Making it mandatory to apply hold time to the sending of messages would potentially make BGP peerings less stable. It clearly can’t make them *more* stable. Of course one can argue that if you haven’t been able to send a message for the hold time, the session has failed its metric of usefulness anyway, so any veneer of stability at this point is a harmful sham.
    >    - If I recall correctly, RST doesn’t work (or may not work) if you’re using the MD5 TCP option. Nothing much to be done, but be aware.
    >    - There is nothing stopping an implementation from doing what you describe now. The formalism that keeps you within the letter of 4271 would be that the implementation supplies a configuration option, that you set to enable the behavior. Once you’ve done that, when the implementation notices that the hold time has been exceeded in the outbound direction, it generates a ManualStop event for the session. 
    > 
    > #Keyur: +1 to what John said. This could very well be an implementation knob that generates ManualStop event.
    > 
    > Regards,
    > Keyur
    > 
    >    Thanks,
    > 
    >    —John
    > 
    >> On Dec 11, 2020, at 2:23 PM, Job Snijders <[e-mail]> wrote:
    >> 
    >> 
    >> Dear group,
    >> 
    >> Not too long ago an incident [1] in one Autonomous System resulted in
    >> the global Internet being unusable in many parts of the world for
    >> multiple hours. Some have reported the root cause was a 'configuration
    >> error', however I believe much of the observed communication blackouts
    >> in the global routing system stemmed from a pre-existing condition: a
    >> specific implementation property present in multiple implementations
    >> currently in use in the default-free zone.
    >> 
    >> Usually when an incident happens in one AS, affected parties can through
    >> unilateral action 'route around the problem', but the ability to 'route
    >> around problems' critically depends on the ability to distribute
    >> WITHDRAW or UPDATE messages. When messages are not processed, what
    >> generally was assumed to be a unilaterally solvable problem, now requires
    >> coordination between *all* neighbors of the suffering AS.
    >> 
    >> The global routing system requires every participant to process BGP
    >> messages, because the alternative is intervention on thousands of BGP
    >> devices to manually shutdown thousands of BGP sessions disconnecting the
    >> AS suffering from an incident, to help the rest of the default-free
    >> zone. I speak from experience when saying that coordinating a disconnection
    >> of an AS at global scale is incredibly hard and slow, any many approval
    >> levels must be worked through. It takes *hours* of phone calls & email
    >> chains, a time window during which internet traffic is routed towards
    >> stale (now blackholing) locations.
    >> 
    >> In the average ISP's network design using IBGP Route Reflectors, these
    >> blackout effects are aggravated when BGP sessions landing in such
    >> devices are not terminated when TCP causes the BGP session to stall.
    >> 
    >> The problem of how TCP and BGP-4 can interact has been discussed before,
    >> but I'm not sure the working group followed up with any publication
    >> detailing the problem and the solution.
    >> 
    >>   https://urldefense.com/v3/__https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkPhCc8cBA$
    >> 
    >> Does everyone agree BGP-4 sessions MUST be terminated using a TCP RST
    >> (instead of a BGP-4 Cease NOTIFICATION) if the peer has indicated for
    >> the duration of the Hold Timer that the TCP receive window is zero?
    >> I'm fine with there being buttons to make this different, but the
    >> default for routers in the global Internet routing system should be to
    >> consider the remote peer to be 'a lost cause' when it won't accept new
    >> BGP messages for the duration of the hold timer.
    >> 
    >> Perhaps RFC 4271 Section 6.5 should be amended as following:
    >> 
    >> OLD:
    >>   If a system does not receive successive KEEPALIVE, UPDATE, and/or
    >>   NOTIFICATION messages within the period specified in the Hold Time
    >>   field of the OPEN message, then the NOTIFICATION message with the
    >>   Hold Timer Expired Error Code is sent and the BGP connection is
    >>   closed.
    >> 
    >> NEW:
    >>   If a system does not receive (or is unable to send) successive
    >>   KEEPALIVE, UPDATE, and/or NOTIFICATION messages within the period
    >>   specified in the Hold Time field of the OPEN message, then the
    >>   NOTIFICATION message with the Hold Timer Expired Error Code is sent
    >>   and the BGP connection is closed. If the NOTIFICATION message cannot
    >>   be send the BGP connection is closed.
    >> 
    >> This is an ongoing problem. I suspect the BGP Nyancat's discoloration at
    >> the left most eye might have been caused by an active TCP session
    >> keeping a stale BGP session alive. But also the observations from "BGP
    >> Zombies: an Analysis of Beacons Stuck Routes" [3] could be explained by
    >> the problematic interaction between TCP and BGP.
    >> 
    >> I appreciate the work the IDR working group has done to *SOFTEN* the
    >> blow from implementation defects on global routing (RFC 7606 is a
    >> brilliant example of this), but I fear in this case there is no subtle
    >> way to say goodbye when the peer doesn't process messages in a timely
    >> fashion. It might be good to document this.
    >> 
    >> Kind regards,
    >> 
    >> Job
    >> 
    >> [1]: https://urldefense.com/v3/__https://www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMkF2w4cg$
    >> [2]: https://urldefense.com/v3/__https://labs.ripe.net/Members/cteusche/bgp-meets-cat__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMry7Ktyw$
    >> [3]: https://urldefense.com/v3/__https://www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkO8A78j8Q$
    >> 
    >> _______________________________________________
    >> Idr mailing list
    >> [e-mail]
    >> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMMXdwc-g$
    > 
    >    _______________________________________________
    >    Idr mailing list
    >    [e-mail]
    >    https://www.ietf.org/mailman/listinfo/idr
    > 
    > _______________________________________________
    > Idr mailing list
    > [e-mail]
    > https://www.ietf.org/mailman/listinfo/idr

_______________________________________________
Idr mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/idr