Re: [Dcrouting] draft-przygienda-rift-03

[Robert Raszuk <[e-mail]>]

Hi Tony,

> ​
if you're willing to provision things correctly yourself via S-PGP.

I am not willing to do that. I would like routing to do it for me
auto-magically. But yes you got the question right. I was asking for
shortcuts between last levels of fabric. Not so much between Node 111 &
Node 112 - but you said it is optional so ok.

Side note: PGP abbrev. for vast majority of people means completely
different thing then what you defined it to mean locally in your draft. I
highly recommended you rename it in -05 version to PGD (policy guided
destination(s) or PGR (policy guided reachability/routing).

Now requirement to switch off miscabling detection is not acceptable. You
are stating that only rift knows how should I cable my fabric ? And if I
cable it some other way it will detect it as miscabling ? I think in most
cases of correct cables check you actually define the intent then network
detects if such intent is met.

> Node112 can actually even go haywire

That may be not best property of a routing protocol :)

Best,
R.


On Sat, Jan 13, 2018 at 7:16 PM, Tony Przygienda <[e-mail]>
wrote:

> So I thought over your horizontal link case to shortcut the spine levels
> for some kind of traffic again and I think the current draft actually
> covers that
> ​​
> if you're willing to provision things correctly yourself via S-PGP.
> Let me see whether we agree on this picture first:
>
> .                +--------+          +--------+
> .                |        |          |        |          ^ N
> .                |Spine 21|          |Spine 22|          |
> .Level 2         ++-+--+-++          ++-+--+-++        <-*-> E/W
> .                 | |  | |            | |  | |           |
> .             P111/2|  |P121          | |  | |         S v
> .                 ^ ^  ^ ^            | |  | |
> .                 | |  | |            | |  | |
> .  +--------------+ |  +-----------+  | |  | +---------------+
> .  |                |    |         |  | |  |                 |
> . South +-----------------------------+ |  |                 ^
> .  |    |           |    |         |    |  |              All TIEs
> .  0/0  0/0        0/0   +-----------------------------+     |
> .  v    v           v              |    |  |           |     |
> .  |    |           +-+    +<-0/0----------+           |     |
> .  |    |             |    |       |    |              |     |
> .+-+----++ optional +-+----++     ++----+-+           ++-----++
> .|       | E/W link |       +=====+       |           |       |
> .|Node111+----------+Node112|     |Node121|           |Node122|
> .+-+---+-+          ++----+-+     +-+---+-+           ++---+--+
> .  |   |             |   South      |   |              |   |
> .  |   +---0/0--->-----+ 0/0        |   +----------------+ |
> . 0/0                | |  |         |                  | | |
> .  |   +---<-0/0-----+ |  v         |   +--------------+ | |
> .  v   |               |  |         |   |                | |
> .+-+---+-+          +--+--+-+     +-+---+-+          +---+-+-+
> .|       |  (L2L)   |       |     |       |  Level 0 |       |
> .|Leaf111~~~~~~~~~~~~Leaf112|     |Leaf121|          |Leaf122|
> .+-+-----+          +-+---+-+     +--+--+-+          +-+-----+
> .  +                  +    \        /   +              +
> .  Prefix111   Prefix112    \      /   Prefix121    Prefix122
> .                          multi-homed
> .                            Prefix
> .+---------- Pod 1 ---------+     +---------- Pod 2 ---------+
>
> I assume here that what you ask for is the following scenario:
> a) POD1 being a compute generating very heavy load towards storage in
>    Prefix121.
> b) traffic from POD1 NOT being balanced through the spines but taking
>    a horizontal link Node112 to Node121 to reach your storage in
> Prefix121
>    to save bandwidth? or delay?
> c) The key to riches is -04 section4.2.5.1
> <https://tools.ietf.org/html/draft-przygienda-rift-04#section-4.2.5.1>.
> Northbound SPF
>
>   in the paragraph "Other south prefixes found when crossing E-W link MAY
> be used IIF". Now, we could make it a MUST (but it's really an
> implementation knob IMO) and what it says is that if you are willing to
> inject an "S-PGP" @ Node121 for Prefix121 it will get flooded to Node112
> and Node112 will have a more specific match than the default in N-SPF. From
> 121 normal RIFT takes over since the normal N-Prefix for Leaf121 kicks in
> on Node121. I assume Node112 policy on the ingress is to not propagate the
> S-PGP south but use it for N-SPF only.
>
>
> Observe that
>
> a) you have to switch off miscabling detection for PoD# on those nodes
> since you are "crossing PoDs illegally"
>
> b) if you want whole Pod#1 to do that you either cable Node111 to Node121
> as well (which will load balance whole Pod#1 towards storage without using
> Spine)  OR you propagate S-PGPs south towards leafs (which will cost you
> leaf FIB of course but make sure ALL traffic to storage goes over Node112
> only).
>
> c) Your forwarding on Node112 can actually even go haywire & load-balance
> some of the traffic to Prefix121 using this S-PGP over Node121 and some
> still using the default route towards spine(which here is a blatant
> violation of LPM of course) and RIFT will work just fine (unless you loop
> yourself to death with PGPs you install) but that of course is a deep
> rathole in itself. I just mention it to show why the "non-looping" design
> is so important and makes for the shortcomings for the SPF on a fabric,
> predicted by the non-directional mesh property that Dijkstra solved in his
> time.
>
> so?
>
> --- tony
>
>
>
>
>
>
> On Thu, Jan 11, 2018 at 10:54 AM, Robert Raszuk <[e-mail]> wrote:
>
>> Hi Tony,
>>
>> Thx for elaborating ...
>>
>> Two small comments:
>>
>> A) SID/SR use case in underlay could be as simple as gracefully taking a
>> fabric node out of service. Not much OPEX needed if your NMS is decent.
>> Otherwise in normal link state I can do overload bit, in BGP number of
>> solutions from shutdown to MED to LP ... depending what is your BGP design.
>> In RIFT how do you do that ? Note that overlay (if such exist) does not
>> help here.
>>
>> B) For horizontal links imagine you have servers with 40 GB ports to TOR.
>> Then you have Nx100 GB from TOR up. You are going to oversubscribe on TOR
>> (servers to fabric) most likely 2:1 .. 3:1 etc. So if I want to
>> interconnect TORs because I do know that servers behind those TORs need to
>> talk to each other in a non blocking fashion _and_ I have spare 100 GB
>> ports on TORs having routing protocol which does not allow me to do that
>> seems pretty limited - wouldn't you agree ?
>>
>> Thx,
>> R.
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Jan 11, 2018 at 6:40 PM, Tony Przygienda <[e-mail]>
>> wrote:
>>
>>> Robert, productive points, thanks for raising them ... I go a bit in
>>> depth
>>>
>>> 1. I saw no _real_ use-cases for SID in DC so far to be frank (once you
>>> run RIFT). The only one that comes up regularly is egress engineering and
>>> that IMO is equivalent to SID=leaf address (which could be a HV address of
>>> course once you have RIFT all way down to server) so really, what's the
>>> point to have a SID? It's probably much smarter to use IBGP & so on overlay
>>> to do this kind of synchronization if needed since labels/SIDs become very
>>> useful in overlay to distinguish lots stuff there like VPNs/services which
>>> you'd carry e.g. in MPLSoUDP. In underlay just use the destination v4/v6
>>> address. Having said that, discussion always to be had if you pay me dinner
>>> ;--) and I know _how_ we can do SIDs in RIFT since I thought it through but
>>> again, no _real_ use case so far. And if your only concern is to "shape
>>> towards a prefix" we have PGP in the draft which doesn't need new silicon
>>> ;-P And then ultimately, yes, if you really, really want a SID per prefix
>>> everywhere then you'll carry  SIDs to everywhere since unicast SIDs are
>>> really just a glorified way to say "I have this non-aggreagable 20 bit IP
>>> host address" which architecturally is a very interesting proposition in
>>> terms of scaling (but then again, no account for taste and RFC1925 clause 3
>>> applies) ...  Your LSDB will be still much smaller, your SPF will be still
>>> simple on leaf in RIFT but your FIB will blow up and anything changing on a
>>> leaf shakes all other leafs (unless you start to run pollicies to control
>>> distribution @ which point in time you start to baby-sit your fabric @ high
>>> OPEX). One of the reasons to do per-prefix SID would be non-ECMP anycast
>>> (where SIDs _are_ in fact usefull) but if you read RIFT draft carefully you
>>> will observe that RIFT can do anycast without need for ECMP, i.e. true
>>> anycast in a sense and with that having anycast SID serves no real purpose
>>> in RIFT and is actually generally much harder to do since you need globally
>>> unique label blocks and so on ...
>>>
>>> 2. Horizontal links on CLOSes are not used that way normally all I saw
>>> since your blocking goes to hell unless you provision some kind of really
>>> massive parallel links between ToRs _and_ understand your load. We _could_
>>> build RIFT that way but you give up balancing through the fabric and
>>> loop-free property in a sense (that's a longish discussion  and scaling
>>> since now you have prefixes showing up all kind of crazy places instead of
>>> default). I see enough demand, we get there ...  Otherwise RFC1925 clause
>>> 10 and 5.
>>>
>>> 3. PS1: Yes, lots of things "could" be done and then we "could" build a
>>> protocol to do that and RFC1925 clause 7 and 8 applies. Such horizontal
>>> links, unless provisioned correctly will pretty much just ruin your
>>> blocking/loss on the fabric is the experience (which the math supports). In
>>> a sense if you know your big flows you can build a specialized topology to
>>> do the optimal distribution (MPLS tunnels anyone ;-) but the point of
>>> fabric is that it's a fabric (i.e. load agnostic, cheap, no OPEX and easily
>>> scalable). Otherwise a good analogy would be that you like to build special
>>> RAM chips for the type of data structures you are storing and we know how
>>> well that scales over time. We know now that within 3-4 years
>>> characteristics of DC flows flip upside down without a sweat when people go
>>> from server/client to microservices, from servers to containers and so on
>>> and so on. So if you can't predict your load all the time you need a
>>> _regular_ topology where _regular_ is more of a mathematical than a
>>> protocol discussion. Fabric analogy of "buy more RAM chips in Fry's and
>>> just stick them in" applies here. So RIFT is done largely to serve a
>>> well-known structure called a "lattice" (with some restrictions) since we
>>> need an "up" and "down". Things like hypercubes, thoroidal meshes and so on
>>> and so on exist but CLOS won for a very good reason in history for that
>>> kind of problems (once you move to NUMA other things win ;-) And if you
>>> know your loads and your can heft the OPEX and you like to play with
>>> protocols generally and if you can support the scale in terms of leaf FIB
>>> sizes, flooding, slower convergence & so on & so on and you run flat IGP on
>>> some kind of stuff that you build that doesn't even have to be regular in
>>> any sense. We spent many years solving THAT problem obviously and doing
>>> something like RIFT to replace normal IGP is of limited interest IMO
>>> (albeit certain aspects having to do with modern implemenation techniques
>>> may get us there one day but it's much less of pressing problem than
>>> solving specialized DC routing well IMO again).
>>>
>>> 3. PS2: RIFT cannot build an "unsupported topology" no matter how you
>>> cable (that's the point of it) or rather we have miscabling detection and
>>> do not form adjacencies when you read the draft carefully. That's your
>>> "flash red light" and it comes included for free with my compliments  ;-)
>>> ... Otherwise RFC1925 clause 10.
>>>
>>> Otherwise, if you have concrete charter points you'd like to add, be
>>> more specific in your asks and we see what the list thinks after ...
>>>
>>> thanks
>>>
>>> --- tony
>>>
>>>
>>> On Thu, Jan 11, 2018 at 1:30 AM, Robert Raszuk <[e-mail]>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have one little question/doubt on scalability point of RIFT ...
>>>>
>>>> Assume that someone would like to signal IPv6 prefix SID for Segment
>>>> Routing in the underlay within RIFT.
>>>>
>>>> Wouldn't it result in amount of protocol state in full analogy to
>>>> massive deaggregation - which as of today is designed to be very careful
>>>> and limited operation only at moments of failure(s) ?
>>>>
>>>> I sort of find it a bit surprising that RIFT draft does not provide
>>>> encoding for SID distribution when it is positioned as an alternative to
>>>> other protocols (IGPs or BGP) which already provide ability to carry all
>>>> types of SIDs.
>>>>
>>>> Cheers,
>>>> Robert.
>>>>
>>>> PS1: Horizontal links which were discussed could be installed to
>>>> offload from fabric transit massive amount of data (ex: storage mirroring)
>>>> directly between leafs or L3 TORs and not to be treated as "backup".
>>>>
>>>> PS2: Restricting any protocol to specific topologies seems like pretty
>>>> slippery slope to me. In any case if protocol does that it should also
>>>> contain self detection mechanism of "unsupported topology" and flash red
>>>> light in any NOC.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Dcrouting mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/dcrouting
>>>>
>>>>
>>>
>>
>