Re: [Idr] TCP & BGP: Some don't send terminate BGP when holdtimer expired, because TCP recv window is 0

[Robert Raszuk <[e-mail]>]

Hi,

Couple of observations/questions:

* I do share some concerns that this will make BGP peering less stable.

* Is the "unable to send" only possible under Window = 0 ? What if there is
a local NIC buffer full and we keep dropping it locally ? If we are going
there perhaps we could say "unable to successfully send" meaning send and
get an ACK for it ?

* The proposal is about reusing the HOLD TIME value to bring BGP down when
you are still receiving keepalives however peer sent ACK for the last
segment indicating zero window - is this right ?

* What happens if our side is stuck and not able to process subsequent ACKs
which potentially increase the window on the peer ?

* Most deployments use BFD to make sure peer is reachable. As BFD is often
offloaded from CPU this is not an indication of health of TCP or BGP path.
But what if some deployments can not use BFD (say not supported by the
peer) ? Then those typically reduce BGP HOLD TIME. Would it not be too
fragile in some cases to bring a TCP down in such cases too fast ? Aren't
we overloading HOLD TIME value here a bit ?

* Assume we bring TCP down ... when does it attempt to go up again ? Are we
ok to bring it up only after manual/script action from such a state ?

* As proposed it seems that the change will affect all AFI/SAFIs using
given single TCP session. Even if perhaps all are perfectly healthy and
running on separate cores. If each SAFI would run on a separate TCP session
this would not have such an impact.

* The change seems applicable to both iBGP and eBGP right ?

In summary the attempt here is to fix application issues by cutting the
transport. Sure half broken transport for whatever reason may not be a good
thing to keep in UP state. Especially when redundancy is in place. But my
main concerns are that we are only trying to focus on a single low level
trigger to detect it.

Wouldn't per AFI/SAFI heartbeat be a better option to detect if a peer's
BGP stack is still up and running fine for all applications ?

Thx,
R.


On Fri, Dec 11, 2020 at 9:04 PM John Scudder <jgs=
[e-mail]> wrote:

> [all hats on]
>
> Hi Job,
>
> Thanks for bringing this up.
>
> To take the liberty of summarizing your wall of text :-) you’re saying
> that you believe BGP should tear down its session if it’s unable to send a
> message for the duration of the hold time.
>
> Given that the conversation last time was inconclusive I think this is a
> good thing for the WG to discuss again. If you want to, you (or someone)
> could turn the idea into a short draft that updates RFC 4271, and we could
> have a WG adoption discussion about it. It might help focus the discussion
> but it’s not mandatory.
>
> I’ll point out a few things to start with —
>
> - Making it mandatory to apply hold time to the sending of messages would
> potentially make BGP peerings less stable. It clearly can’t make them
> *more* stable. Of course one can argue that if you haven’t been able to
> send a message for the hold time, the session has failed its metric of
> usefulness anyway, so any veneer of stability at this point is a harmful
> sham.
> - If I recall correctly, RST doesn’t work (or may not work) if you’re
> using the MD5 TCP option. Nothing much to be done, but be aware.
> - There is nothing stopping an implementation from doing what you describe
> now. The formalism that keeps you within the letter of 4271 would be that
> the implementation supplies a configuration option, that you set to enable
> the behavior. Once you’ve done that, when the implementation notices that
> the hold time has been exceeded in the outbound direction, it generates a
> ManualStop event for the session.
>
> Thanks,
>
> —John
>
> > On Dec 11, 2020, at 2:23 PM, Job Snijders <[e-mail]> wrote:
> >
> >
> > Dear group,
> >
> > Not too long ago an incident [1] in one Autonomous System resulted in
> > the global Internet being unusable in many parts of the world for
> > multiple hours. Some have reported the root cause was a 'configuration
> > error', however I believe much of the observed communication blackouts
> > in the global routing system stemmed from a pre-existing condition: a
> > specific implementation property present in multiple implementations
> > currently in use in the default-free zone.
> >
> > Usually when an incident happens in one AS, affected parties can through
> > unilateral action 'route around the problem', but the ability to 'route
> > around problems' critically depends on the ability to distribute
> > WITHDRAW or UPDATE messages. When messages are not processed, what
> > generally was assumed to be a unilaterally solvable problem, now requires
> > coordination between *all* neighbors of the suffering AS.
> >
> > The global routing system requires every participant to process BGP
> > messages, because the alternative is intervention on thousands of BGP
> > devices to manually shutdown thousands of BGP sessions disconnecting the
> > AS suffering from an incident, to help the rest of the default-free
> > zone. I speak from experience when saying that coordinating a
> disconnection
> > of an AS at global scale is incredibly hard and slow, any many approval
> > levels must be worked through. It takes *hours* of phone calls & email
> > chains, a time window during which internet traffic is routed towards
> > stale (now blackholing) locations.
> >
> > In the average ISP's network design using IBGP Route Reflectors, these
> > blackout effects are aggravated when BGP sessions landing in such
> > devices are not terminated when TCP causes the BGP session to stall.
> >
> > The problem of how TCP and BGP-4 can interact has been discussed before,
> > but I'm not sure the working group followed up with any publication
> > detailing the problem and the solution.
> >
> >
> https://urldefense.com/v3/__https://mailarchive.ietf.org/arch/msg/idr/q0Sx5d3zZjfOmOQ4lO2OZAHh9Lc/__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkPhCc8cBA$
> >
> > Does everyone agree BGP-4 sessions MUST be terminated using a TCP RST
> > (instead of a BGP-4 Cease NOTIFICATION) if the peer has indicated for
> > the duration of the Hold Timer that the TCP receive window is zero?
> > I'm fine with there being buttons to make this different, but the
> > default for routers in the global Internet routing system should be to
> > consider the remote peer to be 'a lost cause' when it won't accept new
> > BGP messages for the duration of the hold timer.
> >
> > Perhaps RFC 4271 Section 6.5 should be amended as following:
> >
> > OLD:
> >    If a system does not receive successive KEEPALIVE, UPDATE, and/or
> >    NOTIFICATION messages within the period specified in the Hold Time
> >    field of the OPEN message, then the NOTIFICATION message with the
> >    Hold Timer Expired Error Code is sent and the BGP connection is
> >    closed.
> >
> > NEW:
> >    If a system does not receive (or is unable to send) successive
> >    KEEPALIVE, UPDATE, and/or NOTIFICATION messages within the period
> >    specified in the Hold Time field of the OPEN message, then the
> >    NOTIFICATION message with the Hold Timer Expired Error Code is sent
> >    and the BGP connection is closed. If the NOTIFICATION message cannot
> >    be send the BGP connection is closed.
> >
> > This is an ongoing problem. I suspect the BGP Nyancat's discoloration at
> > the left most eye might have been caused by an active TCP session
> > keeping a stale BGP session alive. But also the observations from "BGP
> > Zombies: an Analysis of Beacons Stuck Routes" [3] could be explained by
> > the problematic interaction between TCP and BGP.
> >
> > I appreciate the work the IDR working group has done to *SOFTEN* the
> > blow from implementation defects on global routing (RFC 7606 is a
> > brilliant example of this), but I fear in this case there is no subtle
> > way to say goodbye when the peer doesn't process messages in a timely
> > fashion. It might be good to document this.
> >
> > Kind regards,
> >
> > Job
> >
> > [1]:
> https://urldefense.com/v3/__https://www.reuters.com/article/level-3-communi-outages-idUSL2N1CB00C__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMkF2w4cg$
> > [2]:
> https://urldefense.com/v3/__https://labs.ripe.net/Members/cteusche/bgp-meets-cat__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMry7Ktyw$
> > [3]:
> https://urldefense.com/v3/__https://www.iij-ii.co.jp/en/members/romain/pdf/romain_pam2019.pdf__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkO8A78j8Q$
> >
> > _______________________________________________
> > Idr mailing list
> > [e-mail]
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/idr__;!!NEt6yMaO-gk!WnfNFxBMMXzuVhI23_QuKvcPfiG3Jwero3GwHhk0hhH6WNn1W0XWUkMMXdwc-g$
>
> _______________________________________________
> Idr mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/idr
>