Re: [Rift] Version -11 submitted for manual posting since the tracker is confused ...

[Tony Przygienda <[e-mail]>]

good, then my time was well spent ;-)

vast majority of engagements I deal the architecture is
orchestrator/BGP-based overlay over RIFT, i.e. EVPN/L3VPN/k8netes & so on
with future discussions RIFT stretching to the host (it looks like in most
cases the overlay will/is stretching to the host first @ which point in
time the case for underlay stretching to the host becomes compelling,
primarily of course in case of multi-homing).  RIFT per se doesn't care
whether it's VXLAN/MPLSoUDP/SRoUDP (I consider MPLS,SRoUDP goldilocks
technology for TE from the leaf to DCI GW which IMO is the most
useful/pracitcal approach to TE but lots of people do vXLAN obviously for
bunch of reasons) or really anything that is an IP frame (we're IP
switching, right ;-). You get out of it default IP forwarding and that's it
(plus, if you want label binding but since IP fabric silicon is mostly very
simple, not much discussion happened on that ;-)

--- tony

On Sat, Mar 28, 2020 at 12:52 AM Gyan Mishra <[e-mail]> wrote:

> Tony
>
> Thank you for the very detailed replies.  I believe I have a better
> understanding of Rift after reading the draft a few times as well as your
> responses.
>
> I can see now how Rift could be used to extend the data center fabric to
> hypervisor as now the table is only a default route.
>
> Have you had any customers try to use vxlan overlay with Rift or has the
> tendency been in general to run Rift overlay with Rift underlay?
>
> Kind regards
>
> Gyan
>
>
>
> On Wed, Mar 25, 2020 at 9:48 PM Tony Przygienda <[e-mail]>
> wrote:
>
>> Gyan, super, one more read cannot hurt. inline
>>
>> On Wed, Mar 25, 2020 at 4:28 PM Gyan Mishra <[e-mail]>
>> wrote:
>>
>>> Hi Tony
>>>
>>> I finally made it through the doc and overall it reads pretty well.  A
>>> lot to digest in one read or even multiple.
>>>
>>> I see that Rift routing protocol takes best of existing routing
>>> protocols creating a new hybrid style routing protocol.
>>>
>>> As this document is very close to being published I don't see any major
>>> or minor technical issues at this point.
>>>
>>
>> ack
>>
>>
>>> Few comments on the technical aspects of the draft and very minor nits.
>>>
>>> In section 4.2.2 it talks about link neighbor discovery and references
>>> RFC 5549 Advertising IPv4 reachability with IPv6 next hop.
>>>
>>> So lets say if the DC Rift domain is configured to support Dual stack so
>>> both IPv4 & IPv6 NLRI reachability,  and lets say all tenant interfaces are
>>> dual stacked as well - however the L3 inter-links between routers in the
>>> domain only have IPv6 configured on the interfaces so the  peer adjacency
>>> would be only IPv6 across the fabric, however it could carry both IPv4 &
>>> IPv6 NLRI.  Correct?
>>>
>>
>>>
>>>
>>> Would that be recommended and any benefit it doing so then dual stacking
>>> all p2p routed links.
>>>
>>> In this scenario would there be a single topological database or one for
>>> IPv4 & one for IPv6.  If every edge is dual stacked and topological path to
>>> reach and v4 or v6 endpoint is the same is it possible to have a single
>>> consolidated link state / distance vector database for both IPv4 & IPv6.
>>>
>>
>> well, so first, RIFT once it has LL v6 address can forward v4 traffic
>> without any further magic (if silicon supports that) by using v6 nexthops
>> to forward v4. All ZTP.
>>
>> Having said that, RIFT does NOT deal fully with the problem of a
>> substrate that has a disjoint v4 and v6 forwarding capabilities (in a
>> single protocol instance). It could but in practical terms this causes lots
>> of complexity since we would need to assert continuity of forwarding across
>> a single AF, something that a leaf starting a packet cannot guarantee
>> unless it sees the full topology which kills scalability immediately.
>>
>> So practically speaking what I see
>> a) if we want a completely "address free" fabric then v6 ND is enough.
>> This brings up a link on v6 addressing and RIFT allows to advertise whether
>> v4 forwarding on such a link is allowed. Run Bruno's code, it does it
>> straight out ;-) So, with a v6 nexthop v4 or v6 forwards fine. Expectation
>> is , yes, that e'thing will be dual stack for foreseable future & @ certain
>> point e'one will flip to pure v6 (which will happen, we're waiting 20 years
>> now, we'll wait another 20 but IMO it will happen ;-)
>> b) if we _really_ want a partitioned v4 and v6 fabric (i.e. some links
>> just do v6 fwd'ing, some just v4, some both v4/v6) then RIFT allows very
>> easily to run multi-topology (ehem ;-) by instantiating it multiple times
>> (just use different mcast address and/or UDP ports). Then run RIFT twice,
>> once for v4, once for v6.
>> c) an implementation could easily declare links miscabled if the v4/v6
>> forwarding capacity on both sides doesn't match and send that to top of
>> fabric where it's visible immediately.
>> d) a very subtle discussion is faith sharing v4/v6 adjacency. I looked @
>> what folks do & I think that running 2 sessions that are disjoint is
>> detrimental in couple ways hence RIFT runs basically one state machine
>> (albeit it sends LIEs on both AFs). There is a very subtle problem of e.g.
>> v6 LIEs going but all of a sudden v4 LIEs dropping and what the semantics
>> of that is? Silicon died? faulty implementation? laziness ;-) ?  RIFT spec
>> doesn't deal with that. Practically speaking an implementation can react to
>> that but e'body has a different opinion what the semantcis are. Ultimately,
>> one can resolve that by running 2 BFD sessions under RIFT. All very high
>> complexity IMO. RIFT is _not_ BGP that has to keep a session up @  high
>> compleixty of supporting reconfigurations across e.g. AFs. RIFT should be
>> easy to operate & fast, one just breaks adjacency, flap is local, fabric is
>> densely connected and how often _really_ one would want to remove/add AFs
>> on a fabric link?
>>
>> So, in short, for all practical purposes any v6 silicon today can do v4
>> forwarding anyway so we flip on ND and enable 'family inet' on a link which
>> magically does v6/v4 forwarding and then once v6-only silicon dominates I
>> expect e'one will flip whole fabric or worst case v4 forwarding capability
>> is not advertised anymore.
>>
>>
>>>
>>> Can RIFT be used as the IGP in an SP or Enterprise core "mpls
>>> replacment" scenario outside the Data Center paradigm which this is
>>> geared.  If so then eventual IGP extensions and support would follow if
>>> that is a future goal of the WG.
>>>
>>
>> longer answer in another email.
>>
>>
>>> Another use case for RIFT I could see in any hub/spoke type topology
>>> such as  DMVPN hub/spoke topologies where the spokes receive Default
>>> distance vector to the edge and link state at the spine hub.
>>>
>>
>> right ;-)
>>
>>
>>> Correct me if I am wrong but "RIFT" is really just another IGP in the
>>> operators toolbox to be utilized and that use case can literally be
>>> anywhere and not just the DC.
>>>
>>
>> yes, pls read use-cases draft. Main driver for RIFT was actually NOT data
>> center fabric case but I won't say more ;-)
>>
>>
>>> As far as overlay & underlay technologies such as NVO3 vxlan evpn
>>> overlay, Rift as stated in section 4.3.3.4 does not preclude any overlay
>>> protocols.
>>>
>>
>> well, RIFT is pure underlay really. Given how cheap it is to run multiple
>> RIFTs (or if your substrate is v6 only and address space is ___uge ;-) then
>> one could run multiple RIFTs easily and do away with overlay. Some people
>> like that idea and who am I to argue ;-)  To give you even further ideas,
>> some folks want to run RIFT multiple times, once top of fabric being spines
>> & then leafs being spines for a part of fabric being rift "upside down"
>> which has intersting properties. Will that work? yeah, just fine.
>>
>>
>>>
>>> The leaf nodes only get a default route similar to a ISIS level 1 router
>>> with attach bit set. However as in ISIS you can propagate all routes from
>>> Level 2 to Level 1, is it possible for all nodes in the topology to receive
>>> all routes if that is desired.  There maybe some use cases where it is
>>> desirable in case of network partition and black hole scenarios where
>>> having all routes maybe desirable.  In section 6.4 this is addressed
>>> however is it possible to import only intra-area routes and not external
>>> routes redistributed into the topology.
>>>
>>
>> hmm, no. redistribution is kind of orthogonal to protocol spec but you
>> can take any protocol's externals and put it into another protocol again,
>> nothing stops that. RIFT distinguishes external from internal since this is
>> critical for route preference (also on disaggregation). RIFT does not do
>> EXT-1 and EXT-2 and such things since this looks like complexity no'one
>> needs for now (but it could be added).
>>
>> As to the question: can all nodes receive all routes in the topology?
>> Yes, you can run RIFT like that (just flood e'thing e'where) but
>> scalability goes out the window immediately. There is another flavor of
>> that some folks asked to which is to always disagreggate @ bottom of the
>> fabric (i.e. leafs by first level spines) for bunch of reasons and that's
>> not a big deal, one implementaiton knob and the "flat flooding of prefixes"
>> is contained.
>>
>> For more subtle discussion I suggest to read the use cases/applicability
>> draft where some finer points of "how do I get my loopback everwhere?" is
>> explained.
>>
>>
>>>
>>> Does Rift have any MPLS TE FRR like "make before break tunnels" or
>>> TI-LFA pre computed back path support or any SR TE type coloring support?
>>>
>>
>> for MPLS, this is all really overlay stuff, i.e. tunnel run on top of
>> RIFT. RIFT provides unsolicited downstream binding so MPLS will work (or we
>> could run any other label distribution solution over it). For
>> FRR-discussion: RIFT runs on very dense fabrics so we can have an
>> implemenation doing FRR but normally links failing just remove stuff from
>> the W-ECMP next hop on silicon and life goes on with <50msec loss normally.
>> If links southbound fail our path north is basically our FRR, if all links
>> northbound fail we don't have a FRR unless it's southbound which doesn't
>> need FRR ;-) (unless we run horizontal links which will probably be killed
>> immediately by amount of traffic hitting them anyway ;-)  So FRR on fabrics
>> is complex and IMO pretty superfluous, a well written RIFT implementation
>> should fail with sub-second convergence on any single failure. we could
>> argue that we could compute TI-LFA through leaves, that will make traffic
>> on the fabric zig-zag possibly couple times up and down the fabric, that's
>> probably not a very stable solution in terms of behavior ;-)
>>
>> When we talk TE on fabrics (which all the coloring and so on discussions
>> are) then we shall do something once we see demand for that ;-) Last couple
>> years, I simply do not see demand for TE on fabrics and such a thing
>> working _cheaply_ at scale. The goldilocks technology I see deployed and
>> used is MPLSoUDP or SRoUDP where the leaf sticks stuff into UDP that fully
>> utilizes fabric bandwidht and hits the top-of-fabric DC GW (or leaf) that
>> takes MPLS/SR out of it and starts to do TE on the backbone where BW is
>> expensive and TE is needed.
>>
>>
>>
>>>
>>> Does Rift have any Graceful restart capability with redundant RPs?
>>>
>>
>> RIFT is built for fast protocol convergence on failures & easy production
>> in/out and on fabrics where in vast majority of cases we have alternate
>> paths. When going out the fabric, we can set overload bit, wait coiuple
>> 100msecs until the traffic is shifted around and then go away. No need for
>> complex, fabric-wide "drain" operations. Or we can reset adjacencies, that
>> will make the neighbor shift traffic link by link. On bootup we can just
>> make sure that before we start forwarding we built enough routing table
>> state. Not different than ISIS. Could GR be added? Yes, it could but no'one
>> asked and it would add good amount of complexity.
>>
>> NSR/redundant RPs are very expensive and complex technologies I simply
>> don't see on punny fabric IP switches ;-)  Could it be built into RIFT,
>> yes, it could ...
>>
>>
>>>
>>> ISIS is encapsulated directly into L2.  Does RIFT use similar
>>> encapsulated into L2 or does it have an IP protocol number TBD assignment?
>>>
>>
>> I think that's quite elaborately explained. It's all on UDP, long
>> discussion why, baiscally implemenation simplicity on any *nix kernel and
>> very, very easy multi-instancing.
>>
>>
>>>
>>> I see in section 8.1 IANA request for multicast address & port numbers.
>>>
>>> Does Rift use TLVs similar to ISIS to carry LSP PDU information?
>>>
>>
>> nope, that's elaborately explained as well in the spec. RIFT is
>> _completely_ model based (modulo minimal security envelope that cannot be
>> modelled). Advantages of that are vast, refer to very well documnted
>> interop reports where modelling saved us 80% (at least) of interop problems
>> due to buggy parsers, unclear semantics and other oddities with which
>> lovingly crafted ASCII-art formats were keeping us employed over years ;-)
>>
>>
>>>
>>>
>>> 4.3.8 Leaf procedures
>>>
>>> I see that all Leafs don't have a direct link between.  Even pairs of
>>> top of rack leaf nodes don't have a link between them.  Why is that?  I am
>>> guessing convergence and maybe loops.
>>>
>>
>> well, CLOS does not have horizontal links. That puts a serious crimp into
>> blocking prob math and can make traffic run wild over thin links. RIFT
>> supports them due to couple practical oddifites. Below ToF they are last
>> resort northbound, @ ToF level they exchange topology information in case
>> of multi-plane fabrics to prevent unavoidable flooding through the Leafs
>> under certain breakage scenarios. And could be used for forwarding as well
>> if one insists. Spec outlines that. All this is quite elaborately explained
>> in the draft.
>>
>>
>>>
>>> For host redundantly  connected to a pair of leaf node if the leaf level
>>> 1 had pairs of nodes you could re-route locally at the leaf lowest level
>>> then having to go up to the spine east-west link.
>>>
>>> In the topology Leafs have to default route Northbound to the spine e-w
>>> to route leaf-leaf.   So what you are saying is that in case a leaf has an
>>> leaf-leaf interconnect we would set the overload bit on all leaf nodes so
>>> they don't route east-west over leaf-leaf links even if they existed and
>>> are forced to route north bound to the spine east-west.  Would that be sub
>>> optimal then routing to your adjacent leaf node pair.
>>>
>>
>> I kind of lost the directions here but spec treats that. We only allow
>> leaf talk to another leaf, a leaf will never be transit (think what happens
>> if leafs are servers)
>>
>>
>>>
>>>                ^ N      +--------+          +--------+
>>> Level 2        |        |ToF   21|          |ToF   22|
>>>            E <-*-> W    ++-+--+-++          ++-+--+-++
>>>                |         | |  | |            | |  | |
>>>              S v      P111/2  P121/2         | |  | |
>>>                          ^ ^  ^ ^            | |  | |
>>>                          | |  | |            | |  | |
>>>           +--------------+ |  +-----------+  | |  | +---------------+
>>>           |                |    |         |  | |  |                 |
>>>          South +-----------------------------+ |  |                 ^
>>>           |    |           |    |         |    |  |             All TIEs
>>>           0/0  0/0        0/0   +-----------------------------+     |
>>>           v    v           v              |    |  |           |     |
>>>           |    |           +-+    +<-0/0----------+           |     |
>>>           |    |             |    |       |    |              |     |
>>>         +-+----++ optional +-+----++     ++----+-+           ++-----++
>>> Level 1 |       | E/W link |       |     |       |           |       |
>>>         |Spin111+----------+Spin112|     |Spin121|           |Spin122|
>>>         +-+---+-+          ++----+-+     +-+---+-+           ++---+--+
>>>           |   |             |   South      |   |              |   |
>>>           |   +---0/0--->-----+ 0/0        |   +----------------+ |
>>>          0/0                | |  |         |                  | | |
>>>           |   +---<-0/0-----+ |  v         |   +--------------+ | |
>>>           v   |               |  |         |   |                | |
>>>         +-+---+-+          +--+--+-+     +-+---+-+          +---+-+-+
>>> Level 0 |       |  (L2L)   |       |     |       |          |       |
>>>         |Leaf111+~~~~~~~~~~+Leaf112|     |Leaf121|          |Leaf122|
>>>         +-+-----+          +-+---+-+     +--+--+-+          +-+-----+
>>>           +                  +    \        /   +              +
>>>           Prefix111   Prefix112    \      /   Prefix121    Prefix122
>>>                                   multi-homed
>>>                                     Prefix
>>>         +---------- PoD 1 ---------+     +---------- PoD 2 ---------+
>>>
>>>
>>>               Figure 2: A Three Level Spine-and-Leaf Topology
>>>
>>>
>>> Nits
>>>
>>> 4.3.1 overload bit - Someone familiar with ISIS would understand use.  Maybe add a description and also if the behavior of the overload bit used with Rift is any different from ISIS.
>>>
>>>
>> no difference, really a complete knock-off ;-) overlaod bit is one of the
>> best things designed into an IGP, ever ;-)
>>
>>
>>> 4.3.7 Label binding
>>>
>>>    A node MAY advertise in its LIEs, a locally significant, downstream
>>>    assigned, interface specific label.  One use of such a label is a
>>>    hop-by-hop encapsulation allowing forwarding planes to be easily
>>>
>>>      distinguished among multiple RIFT instances.
>>>
>>> I didn't see anywhere in the draft but maybe a diagram of multiple
>>> dimensional 2-D 3-D forwarding planes instances where the label binding is
>>> used as the distinguisher.
>>>
>>>
>>
>>> Is the concept similar to a 64 bit RD making a prefix unique within a VRF in an L3 VPN context?
>>>
>>>
>> well, no, think more unsolicited downstream LDP. kind of nice, gives you
>> a label you can run SR/MPLS/do whatever (just like node/adj SIDs are used
>> in SR these days often). Outside base RIFT spec how that's used then. Just
>> think that you'd multi-instantiate RIFT over different mcast and/or ports &
>> each adjacency gives you a different label. Pronto,
>> multi-planes/multi-topology/whatever ...
>>
>> uff, longish mail ...
>>
>> -- tony
>>
>>
>>
>>>
>
> --
>
> Gyan  Mishra
>
> Network Engineering & Technology
>
> Verizon
>
> Silver Spring, MD 20904
>
> Phone: [phone]
>
> Email: [e-mail]
>
>
>
>