[Rift] Why negative disaggregation cannot replace positive disaggregation in all cases.

[Bruno Rijsman <[e-mail]>]

Hi Pascal,

Yes, we can do a monthly.

For now, here is the issue that I ran into that makes me believe that negative disaggregation cannot replace positive disaggregation in all cases (specifically not for the “normal” case of non-tof routers).

Here is a simplified summary of how positive disaggregation works (see section 4.2.5.1 for details):

If this router R1 has a "direct route" for prefix P, and that route P has a set of next-hop {NH}, and there is some other router R2 at the same level as R1 that does not have NH as a south-adjacency for each NH in {NH}, then this router R1 must positively disaggregate prefix P: this router R1 must "attract" the traffic so that it doesn't go to the other router R2 (who cannot delivery the traffic to P).

Now let's try to the the reverse thing with negative disaggregation:

If some other router R2 at the same level as R1, and R2 has a "direct route" for prefix P, and that route P has some set of next-hops {NH}, and this router R1 does not have NH as a south-adjacency for each NH in {NH}, then this router R1 must negatively disaggregate prefix P: this router R1 must "repel" the traffic so that it goes to R2 instead of R1.

The problem that this requires router R1 to “ now” what the set of routes is that router R2 has, and to know the next-hops for those routes. But R1 cannot know/calculate the "direct routes" that R2 has because it doesn't know which north-TIEs R2 has in its database. So, I don’t think we can do negative disaggregation here...

Am I correct or am I missing something?

— Bruno


On May 18, 2020, at 12:28 AM, Antoni Przygienda <[e-mail]<mailto:[e-mail]>> wrote:

Agreed. Let’s set up a monthly


  *   Tony


From: "Pascal Thubert (pthubert)" <pthubert=[e-mail]<mailto:pthubert=[e-mail]>>
Date: Sunday, May 17, 2020 at 10:15 PM
To: Bruno Rijsman <[e-mail]<mailto:[e-mail]>>
Cc: Antoni Przygienda <[e-mail]<mailto:[e-mail]>>, "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>, Tony Przygienda <[e-mail]<mailto:[e-mail]>>
Subject: Re: [Rift] Negative disaggregation feature guide

[External Email. Be cautious of content]

This is quite interesting, Bruno.

Does that mean that there are issues running negative in stand alone? I’d be really interested if we have a monthly or something one of these days. Also there’s the section on disaggregation in the applicability draft that we may enrich
Many thanks for all your efforts!

Pascal


Le 18 mai 2020 à 01:02, Bruno Rijsman <[e-mail]<mailto:[e-mail]>> a écrit :
Hi Tony,

I ran into various complications and a road-block while working on the “disaggregation negative-only” knob in RIFT-python, so I will be ripping that knob out of my code.

Instead I will stick to the three commandments as handed down on the stone tablet by the wise RIFT gods from above :-)


On May 14, 2020, at 12:26 PM, Antoni Przygienda <prz=[e-mail]<mailto:prz=[e-mail]>> wrote:

In a probably more detailed way with a little stone tablet I suggest 😉


  1.  You SHALL negatively disaggregate only if you’re ToF and have horizontal links (ring)
  2.  You SHALL propagate transitively negative disaggregation
  3.  You SHALL use positive if you’re not ToF or have no horizontal


Of course, any vendor, even open sourced one can add any flavor of knobs that makes their dish unique even if it breaks the spec strictly speaking … So did I 😉

--- tony

From: Tony Przygienda <[e-mail]<mailto:[e-mail]>>
Date: Thursday, May 14, 2020 at 11:12 AM
To: Bruno Rijsman <[e-mail]<mailto:[e-mail]>>
Cc: "Pascal Thubert (pthubert)" <[e-mail]<mailto:[e-mail]>>, "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>, Melchior Aelmans <[e-mail]<mailto:[e-mail]>>, Christian Graf <[e-mail]<mailto:[e-mail]>>, Oliver Steudler <[e-mail]<mailto:[e-mail]>>, Olivier Vandezande <[e-mail]<mailto:[e-mail]>>, Antoni Przygienda <[e-mail]<mailto:[e-mail]>>, "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>, Jeff Tantsura <[e-mail]<mailto:[e-mail]>>, Zhaohui Zhang <[e-mail]<mailto:[e-mail]>>, Zhaohui Zhang <[e-mail]<mailto:[e-mail]>>, Jeffrey Zhang <[e-mail]<mailto:[e-mail]>>
Subject: Re: Negative disaggregation feature guide

[External Email. Be cautious of content]

 my thoughts:

negative is seriously more complex to  implement and understand operationally and only needed on multi-plane fabrics, taht's why positive is KISS ;-) Also, negative always forces you to ring the top of the fabric which in single plane design is an unnecessary requirement.

I foresee vendors not implementing negative for simplicity, very small footprint fabrics.

the discussion about capability advertisement is interesting, we can just add one bit on the node capabilities element, make it optional and default true.

Of course an implementation taht understands that all involved nodes understand negative is free to use negative instead of positive then.

-- tony

On Thu, May 14, 2020 at 11:05 AM Bruno Rijsman <[e-mail]<mailto:[e-mail]>> wrote:
Broadening a discussion on negative disaggregation to the RIFT mailing list.

Now that we have implemented negative disaggregation in RIFT-Python, and now that we are getting some operational experience with it, the following is becoming more and more evident:

It seems to us (Bruno and Pascal, for now) that once negative disaggregation is implemented, there is really no need for positive disaggregation any more.

Every use case that can be solved by positive disaggregation can (as far as we can currently tell) also be solved by negative disaggregation.

Furthermore, negative disaggregation solves the problems in a far simpler and more elegant way: there are fewer advertising nodes and fewer advertised TIEs involved, and we don’t have the synchronization issue that positive disaggregation has that potentially causes incast problems.

Thus, at the very least, it makes sense to recommend that in a given fabric we use either only negative disaggregation only or positive disaggregation only, with negative disaggregation being the default.

If positive and negative disaggregation are enabled simultaneously in one and the same fabric, everything still works fine as far as we can see. But there are some “interesting” interactions that make things unnecessarily complex and potentially fragile.

If support for negative disaggregation is mandatory, and if our assessment that negative disaggregation can solve all use cases is correct, then we could go one bold step further and completely remove positive disaggregation from the specification.

Thoughts?

— Bruno



On May 14, 2020, at 10:48 AM, Pascal Thubert (pthubert) <[e-mail]<mailto:[e-mail]>> wrote:

Hello Bruno:

I like your negative-only default because it is probably the safest. Let me elaborate..

partitioning saves a lot more ports per ToF node than the ring cost so it does not matter.

If you enable negative, there’s no point in doing any positive at all, is there? So what do we try to save by combining? In your example, I guess that super 1 2 had to retract the positive disag and then do negative instead when the link to spine 1 1 fell. Was that not a bit complex and error prone?

Also I’m not clear how we can always decide that dynamically and I do not believe that we specified that. So I understand that it’s more like a use case thingy, like a configuration that would be adapted to the use case.

=> do not configure both at the same time use either positive or negative disag.

We do not try to do positive disag transitively. So if there is a need for transitive, you have to use negative.  In a multiplane case, there’s usually (as in your picture) one ToP per plane in each PoD, and a first link failure (say Spine 1 1 to leaf 1 1) can already cause a fallen leaf: leaf 1 1 is no more reachable within plane 1. This tells you that all leaves in the other PoDs must avoid that plane, and that’s a leaf decision; which tells you that you need to recurse transitively down. This indicates that negative must be activated in a highly partitioned (meaning low redundancy) multiplane.

Say we try to use positive disaggregation transitively in a ToF that is partitioned like in planes, you’d run a risk that none of the ToF nodes that can still reach the leaf can see (through south reflection) the ToF node that cannot so they do not know they need to disag. So you end up needing the same ring as for negative, the southern reflection become mostly useless and the benefits of positive are gone.

Partitionned ToF => negative

I trust that we can use positive in a highly redundant and symmetrical non-partitioned ToF. If no one can reach the leaf anymore (like it’s dead) we have no issue since it should not be disaggregated, just use the discards route at any ToF node. If you have enough redundancy and no partition,  there will always be nodes that can both reach a leaf and discover that a peer does not.

=> Positive is indicated for a very specific use case, ideally 2 levels and fully meshed. Maybe that’s enough deployments to justify the feature in the code.

So the question is really that blurry zone between highly redundant single plane and highly partitioned multiplane what should one do? I becomes a risk/benefit judgement… If negative is implemented and the ring is there, I’d use it, better safe than sorry.

Take care

Pascal



From: Bruno Rijsman <[e-mail]<mailto:[e-mail]>>
Sent: jeudi 14 mai 2020 17:36
To: Pascal Thubert (pthubert) <[e-mail]<mailto:[e-mail]>>
Cc: Melchior Aelmans <[e-mail]<mailto:[e-mail]>>; Christian Graf <[e-mail]<mailto:[e-mail]>>; Oliver Steudler <[e-mail]<mailto:[e-mail]>>; Olivier Vandezande <[e-mail]<mailto:[e-mail]>>; Tony Przygienda <[e-mail]<mailto:[e-mail]>>; Antoni Przygienda <prz=[e-mail]<mailto:prz=[e-mail]>>; [e-mail]<mailto:[e-mail]>; Jeff Tantsura <[e-mail]<mailto:[e-mail]>>; Jeffrey (Zhaohui) Zhang <[e-mail]<mailto:[e-mail]>>; Jeffrey (Zhaohui) Zhang <zzhang=[e-mail]<mailto:zzhang=40juniper.net@dmarc..ietf.org>>; Jeffrey Zhang <[e-mail]<mailto:[e-mail]>>
Subject: Re: Negative disaggregation feature guide


On May 14, 2020, at 8:53 AM, Pascal Thubert (pthubert) <[e-mail]<mailto:[e-mail]>> wrote:

This does not show because on your first breakage you use the positive disag. If you stick to the logic you used at the beginning, that is pick an example where positive applies but use negative, then on your first breakage you’d show that the negative does not need to go transitively to the leaf, because the spine nodes still have solutions. Is that a lot of work to change?

This is exactly what I was thinking as well.

Maybe I need to go through the current specification with a fine-toothed comb again, but my (possibly incorrect) understanding is that the current version of the specification is that we positive disaggregation is used (MUST be used? SHOULD be used?) in the “first failure” scenario.

Personally, I see no reason why we would not be able to use negative disaggregation instead of positive disaggregation in the first failure scenario as well.

In the context of the tutorial, this would have the advantage of clearly demonstrating the propagation logic more clearly.

But more importantly, it seems to me that once negative disaggregation is implemented, there is really no reason to use positive disaggregation anywhere. It would be “cleaner” (less signaling) and “better” (no incast) to simply use negative disaggregation everywhere.

So…. I was thinking to add a configuration knob to my code “positive-only”, “negative-only”, “positive-and-negative”, with “negative-only” being the default? (Seems like a waste to have coded up the positive disaggregation).

What does everyone think of that?

Should we go even further and simply remove positive disaggregation from the spec?

Should we have negative disaggregation as a capability in the capability negotiation? (Seems non-sensical, given it is something that the whole fabric needs to support or not.)

Should we take this to the RIFT mailing list?

— Bruno

_______________________________________________
RIFT mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/rift<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Frift__%3B!!NEt6yMaO-gk!TmsyrEr6P-LonGpbG3vC0JFQ_i21Ign6Y6BE9wvWBuAioJWe1MkV9Q-fUBH1Rg%24&data=02%7C01%7C%7C4db24006111d4624c40708d7faf4bcef%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637253801557732691&sdata=R8MLL0Q8aTmhr4xq%2BGTTTf2IgKLWcDoBKQSz2APcGsI%3D&reserved=0>

_______________________________________________
RIFT mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/rift


Juniper Business Use Only