OT and trimmed (was Re: Q on the congestion awareness of routing protocols)

[Curtis Villamizar <[e-mail]>]

In message <[e-mail]>
"Bless, Roland (TM)" writes:
 
> Hi Curtis,
>  
> On 04.12.22 at 12:06 Curtis Villamizar wrote:
> > Is it my imagination or did this conversation start out about PIM-SM
> > and wander off into everything but PIM-SM?
>  
> I think that Toerless question was about congestion control for
> routing protocols in general and multicast routing in particular.
> Thanks for pointing out that specific bottleneck on updating
> the forwarding plane (Toerless also pointed at that) and the
> work-conserving property of BGP.

Not assigning blame for thread wander.  Toerless was commenting on
PIM-SM with the open question of whether other protocols (mostly focus
on multcast I think) had similar congestion issues.

> > Anyway ... regarding BGP and TCP (or SRM).
>  
> [... trimmed ...]
>  
> > This conversation did not start out being about delay based routing.
> > We all know that any use of a varrying route metric is bad without
> > some mechanism to insure stability (hopefully we all know ..).
>  
> I guess it was Stuart Bryant who mentioned shortest time routing,
> and therefore I briefly mentioned it here...

Yeah Stewart.  Shame on you.  [Now we are asigning blame.]  :-)

> >> I actually have not the operational experience as others
> >> may have, but my guess is that practically CPU congestion
> >> occurs more often than link congestion (solely caused by
> >> control plane packets). While I believe that TCP congestion control
> >> may potentially help to fix short-term congestion situations,
> >> it is not a solution for persistent link congestion – I think that
> >> such a system may not be able to function correctly.
> > 
> > The bottlenect in core routers is actually installing the routes onto
> > the forwarding cards.  Multiple cards to talk to with less processing
> > power in some cases.  Congestion control to the forwarding control
> > plane goes back to the route socket BSD days in early to mid 1990s but
> > installing on separate smart forwarding cards is now more often in
> > application space.
> > 
> >> So there are typically dampening mechanisms in place to aggregate
> >> routing information or to wait before announcing certain
> >> route updates.
> >> When using TCP, the CPU congestion problem would cause
> >> flow control to kick in and automatically throttle
> >> the sender to the receiver's processing speed.
> >> However, if the generation rate of routing messages is
> >> permanently too high, the system will not be stable.
> > 
> > Last sentence is not true.  The work conserving aspect of BGP
> > implementation (any that have survived and are used) means you have
> > one most recent route for any destination and are throwing away old
> > routes before ever installing them.  Routes get installed in the order
> > of the least recently received route and if changed before getting
> > installed goes to the back of the queue.  Similar work conserving can
> > be done at Adj-In and is useful if BGP Adj-In is delegated to
> > forwarding cards with reliability built into TCP state.  Multiple
> > routes arrive with each TCP packet, particularly with large MTU, so
> > acks get delayed a bit to allow for failure of the forwarding card a
> > bit like database 2-phase commit.  The routing table install queue is
> > finite and each BGP Adj-In queue is finite.  Both are bound by the
> > number of prefixes in global routing so to protect from getting
> > flooded with bad prefixes it helps to have some sanity check to avoid
> > disaggregation or bogus prefixes.  Any bottleneck in route table
> > install, IGP SPF, or forwarding install does not slow down BGP Adj-In
> > processing or TCP transfer of information.
>  
> The precondition for my statement was CPU congestion
> (in your case: BGP Adj-In processing is tool slow).
> If I understand correctly you say that there is a decoupling between
> routing message processing and FIB entry installation and
> your assumption is that neither BGP Adj-In processing
> nor TCP processing are a bottleneck. However, I didn't
> fully understand the part where "BGP Adj-In is delegated to
> forwarding cards with reliability built into TCP state"
> and "acks get delayed a bit to allow for failure of the forwarding
> card". This sounds like an approach with tight coupling
> between FIB entry installation and TCP processing.
> So in case of failure (what kind of failure? malfunction of the
> whole card or just failure of the installation of entries?)
> the ACK is not sent?
> Then the sender retransmits and the installation is tried again?

The Adj-In on forwarding cards was done by Avici after they did their
"Non-Stop Routing" which was trademarked and patented (problems with
the patent imnsho).  The trademark was withdrawn.  NSR was failover if
a route processor card failed and required a kernel hack to backup the
TCP state on any secondary route processor among other things.  When
Adj-In was moved to forwarding cards the same was done there so that
if either the link failed and packets arived at another card or the
card itself failed another card could take over and the other end
would never know the difference.  [Quite off topic.]

I'm not sure how much of the NSR capability spilled over to other
router vendors in the following two decades.  NSR is definitely off
topic though an interesting topic.

> Regards,
>   Roland
>  
> > Regarding "it helps to remember" from the slides from Ross and John it
> > helps to remember BGP work conserving nature that was discussed
> > extensively in the mid 1990s but still gets overlooked now and then.
> > 
> > Now back to PIM-SM ... maybe.

Or maybe not.

Curtis