Re: [Rift] RIFT

[Antoni Przygienda <[e-mail]>]

Hey, Kris, inline

________________________________
From: Kris Price <[e-mail]>
Sent: Thursday, April 18, 2019 10:28 AM
To: Antoni Przygienda
Cc: [e-mail]; [e-mail]
Subject: Re: RIFT

Hey Tony,

On the rings: Ahh! I get it, okay that makes it better. I was also
wondering if some kind of designated 'S-TIE' reflector / virtual links
/ or explicitly configured multi-hop adjacencies solution could be
used (the issue being one of how do you route these packets between
the peers without needing to do something like source route multiple
hops southbound before being default routed northbound).

good, I know it takes bit to grok the stuff. We did the best we could with ASCII and language but the concepts need some chewing for sure, even if you have been around big fabrics for a bit ;-) So, nothing like route reflectors and so on, within a plane normal south reflection takes care of sync'ing up all you need, outside the plane the ring takes care of sync'ing up planes (for flooding horizontal links below ToF are south and @ ToF level north basically and with that you have all the topology to figure out negative disaggregation.  I explicitly killed any "virtual link" suggestions, I went through this particular hell in my life more than once and don't want to visit it anymore ;-) ...

Back on the subject of disaggregation:

The other reason for asking for the always disaggregate option is to
prevent the transient congestion that can occur on link failures. But
I do see now on rereading the draft you've called this out in the
second to last paragraph of 5.2.5.1., but it's left it as an
implementation specific problem to solve.

well, yes, no free lunch, either you gum up your fabric with all stuff and suffer large blast radius or you dig the beauty of having minimum blast radius and minimal topology info everywhere but on massive failures stuff needs to be sloshed around so e'one has enough info to not blackhole. Finely enough, today's networks, especially fabrics, allow insane flooding rates without breaking half a sweat (first thing I played with when thinking about RIFT design ;-) and I learned here some lessons from looking @ p2p networks BTW. If you run my free package you'll see easily convergence rate of 7-10+K TIEs in the database per second and that's the "usable rate" in the sense that there is much more flooding on the links and it's the "best TIEs in LSDB" rate already. UDP is really quite phenomenal and with a bit of additional help (look @ the packet number & the "you flood too fast" indications) you can dynamically adjust flooding to walk the edge of having losses. Brave new world ...

It seems this would arise frequently at the bottom two tiers of the
network. Any loss of any single link to any rack (tier 0) would result
in all other nodes at tier 1 disaggregating the prefix(es) for that
rack and causing the potential transient incast-like congestion. I'm a
bit concerned that this may be a noticeable event in some cases (e.g.
a storage row/cluster or maybe where RoCE is in use), and one that
would be fairly annoying to debug and remedy post transition to RIFT
if you didn't foresee it and have the tools (knobs) in place to
prevent it from happening without a PR and s/w upgrade.

yepp, you call the spade but you're a bit too pesimistic me thinks. Let's assume 2 ToRs dual-homing a rack or couple racks of servers. if you loose a link in a multi-homed server you basically end up having the other ToR de-aggregating just this server prefix to other servers (even if you run some kubernetes @ scale you may have 100 prefixes or so I'd say, I can't imagine a server hosting thousands really) ... Then, if you think about the ToRs on top of PoD then it's not as bad as you think. If you loose a single ToR in a PoD towards a spine (I'm loose with terminology here) then you will NOT see disaggregation as long the other ToRs in the PoD are still connected to the PoD. Draw pictures & run the public consumption package ;-)  More interesting discussions are bandwdith balancing on link losses (which I think we solved well northbound) and whether it even should be done southbound since notion of "available bandwidth southbound" is confounding ... Spec doesn't forbid it (the beauty of loop-free valley-free routing that gives you insane amount of lee-way how you choose to forward) BTW if somone is smart enough to figure that out ;-) ...

Should
implementations have a conscious solution in advance for this, and
what's the best way to ensure that? The 'always-disaggregate' knob is
one. Another might be something like a 'min-next-hops' option where
the local RIFT instance on tier 0 won't install a prefix unless it has
received it from a minimum number of up streams

The always disaggregate knob is something you can do per level if you desire but it's basically a big hammer buying you much bigger blast radius in normal operation. And if you pull RIFT onto servers in multi-plane fabrics your FIB may blow up if you do that (unless we think server adapters with 2M FIB size, probably ain't gonna happen ;-).

The other idea I don't grok, you have to explain in more detail.

Both of these do run counter to the low-configuration nature of RIFT.
Another might be a protocol change, something like nodes
disaggregating prefixes by default until they know they are more than
1 hop from the bottom of fabric? (This may run into other convergence
issues during fabric bring up and cold start and maybe there are other
issues with it that need doodling out.)

Yeah, doodle, I'm not concerned about the convergence and interested in your ideas. ZTP has no beef with prefixes, will work irregardless. So far I saw no indications that size of the fabric up to any reasonable bound of nodes will prevent it to cold-boot properly. ZTP FSM has no timers for that reason BTW (another no, no I put down there ;-) and flooding only has a single retransmit timer.

And, yes, the more stuff like forced disaggregation you start to fiddle with, the more you loose ZTP (depending whether you want it or not, RIFT will work either way). BTW, same with security, the more security you desire the less ZTP you get ;-)  Nature of the beast ...

--- tony