Re: [Rift] RIFT

[Kris Price <[e-mail]>]

Hey Tony,

On the rings: Ahh! I get it, okay that makes it better. I was also
wondering if some kind of designated 'S-TIE' reflector / virtual links
/ or explicitly configured multi-hop adjacencies solution could be
used (the issue being one of how do you route these packets between
the peers without needing to do something like source route multiple
hops southbound before being default routed northbound).

Back on the subject of disaggregation:

The other reason for asking for the always disaggregate option is to
prevent the transient congestion that can occur on link failures. But
I do see now on rereading the draft you've called this out in the
second to last paragraph of 5.2.5.1., but it's left it as an
implementation specific problem to solve.

It seems this would arise frequently at the bottom two tiers of the
network. Any loss of any single link to any rack (tier 0) would result
in all other nodes at tier 1 disaggregating the prefix(es) for that
rack and causing the potential transient incast-like congestion. I'm a
bit concerned that this may be a noticeable event in some cases (e.g.
a storage row/cluster or maybe where RoCE is in use), and one that
would be fairly annoying to debug and remedy post transition to RIFT
if you didn't foresee it and have the tools (knobs) in place to
prevent it from happening without a PR and s/w upgrade. Should
implementations have a conscious solution in advance for this, and
what's the best way to ensure that? The 'always-disaggregate' knob is
one. Another might be something like a 'min-next-hops' option where
the local RIFT instance on tier 0 won't install a prefix unless it has
received it from a minimum number of up streams.

Both of these do run counter to the low-configuration nature of RIFT.
Another might be a protocol change, something like nodes
disaggregating prefixes by default until they know they are more than
1 hop from the bottom of fabric? (This may run into other convergence
issues during fabric bring up and cold start and maybe there are other
issues with it that need doodling out.)

/2c

Cheers :-)
Kris


On Fri, Apr 12, 2019 at 1:44 PM Antoni Przygienda <[e-mail]> wrote:
>
> Hey Kris, great to see you engaging back ;-) I cc: rift mailing list for posterity
>
>
> 1) yepp, very nice python implementation, especially if one wants to understand RIFT as running protocol rather than paper spec ;-)
> 2) Yepp, multi-planes did lead to lots of discussions in core-team meetings around acceptable solutions and how we'd explain it properly. Explanation largely due to Pascal's and Ilya's work, it takes a bit to soak the ASCII pictures but once you start to grok the concept of "crossbars crossbaring crossbars" being Clos ;-) it's very easy to think through the stuff.
> 3) A knob to basically "always disaggregate southbound" is  as simple thing to do really. Just like Bruno has it in his computation first phase mostly decides _which_ nodes need disaggregation. The result can be simply replaced by all southbound nodes & then disaggregation happens naturally. Observe that you still want the default origination since a PoD doesn't see other PoDs except via spine and disaggregation is _not_ transitive (I'm talking positive now). There are other cases you want to advertise southobuond some prefixes beside default and it's a normal thing to do, nothing says that you only advertise default southbound anywhere.
> 4) Observe that we do NOT have a single ring! A ring is only as long as the #planes you have. No'one will have 1000 planes ;-) So let's say you have 64 switches in each plane and 4 planes. You will have 64 rings of lengh 4. Obviously you can double-ring or ring within the plane as well to improve reliability but basically the topology is coherent until 2! links in the same ring break.
> 5) Always enabling disaggregation: That's point 3. Observe that does NOT solve your multi-plane problem on breakages since positive disaggregation is NOT transitive. Yes, you can turn southbound PGP on and blast whole fabric with all prefixes which basically makes your blast radius uncontained on any change (kind of flat IGP or rather IGP with DV prefixes ;-) and a single link coming/going may lead to massive amount of convergence traffic due to prefix reachability changes. Moreover, all your leafs (which is servers in extreme case) need FIB size of the size of fabric host routes ...
>
> Spec authorship is still open ;-) so if you feel like improving/adding to draft, just let me know your moniker on bitbucket since thath's where the newest spec versions live ...
>
> https://bitbucket.org/riftrfc/rift_draft/src/master/
>
> --- tony
>
>
> ________________________________
> From: Kris Price <[e-mail]>
> Sent: Friday, April 12, 2019 10:22 AM
> To: Antoni Przygienda; [e-mail]
> Subject: RIFT
>
> Hey guys,
>
> ... couple of thoughts for you:
>
> I see someone much better at explaining things than me talked to you since we last spoke :-) so you've caveated the case with how middle tiers in a Clos don't get full visibility of all nodes at that tier via the reflection because by very nature of the Clos they're not fully striped to that lower tier (that is the essence of how Clos topologies scale so is present in any large fabric). You've covered that under the description as "hyper planes" and work around is to use a ring at those tiers to pass control plane traffic.
>
> Would it be possible to somehow /safely/ have a 'knob' to turn off the aggregation at a tier so it'll always advertise all prefixes southbound? Particularly this is useful in a network that might migrate to this protocol that does not want to go back and cable a ring. And in some cases cabling a ring will be undesirable to some potential users anyway. (In some real world networks that could mean cabling a ring connecting >1000 switches that form the "spine columns". That’s a long brittle ring.)
>
> Larger question, would it be possible to disable aggregation network wide? (For someone that might be interested in using RIFT, but for reasons other than the aggregation capability and where that capability may be seen as undesirable.)
>
> There's a semi-related issue at the very bottom of the fabric but it's a bit difficult to explain, and I'm not sure it's really RIFTs problem to solve, (in part really its due to a bad network design choice that exists) so I might draw that up later.
>
> Cheers
> Kris
>
>
>
>