Re: [Rift] Negative disaggregation feature guide

[Bruno Rijsman <[e-mail]>]

Hi Tony,

I ran into various complications and a road-block while working on the “disaggregation negative-only” knob in RIFT-python, so I will be ripping that knob out of my code.

Instead I will stick to the three commandments as handed down on the stone tablet by the wise RIFT gods from above :-)

On May 14, 2020, at 12:26 PM, Antoni Przygienda <prz=[e-mail]<mailto:prz=[e-mail]>> wrote:

In a probably more detailed way with a little stone tablet I suggest 😉


  1.  You SHALL negatively disaggregate only if you’re ToF and have horizontal links (ring)
  2.  You SHALL propagate transitively negative disaggregation
  3.  You SHALL use positive if you’re not ToF or have no horizontal


Of course, any vendor, even open sourced one can add any flavor of knobs that makes their dish unique even if it breaks the spec strictly speaking … So did I 😉

--- tony

From: Tony Przygienda <[e-mail]<mailto:[e-mail]>>
Date: Thursday, May 14, 2020 at 11:12 AM
To: Bruno Rijsman <[e-mail]<mailto:[e-mail]>>
Cc: "Pascal Thubert (pthubert)" <[e-mail]<mailto:[e-mail]>>, "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>, Melchior Aelmans <[e-mail]<mailto:[e-mail]>>, Christian Graf <[e-mail]<mailto:[e-mail]>>, Oliver Steudler <[e-mail]<mailto:[e-mail]>>, Olivier Vandezande <[e-mail]<mailto:[e-mail]>>, Antoni Przygienda <[e-mail]<mailto:[e-mail]>>, "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>, Jeff Tantsura <[e-mail]<mailto:[e-mail]>>, Zhaohui Zhang <[e-mail]<mailto:[e-mail]>>, Zhaohui Zhang <[e-mail]<mailto:[e-mail]>>, Jeffrey Zhang <[e-mail]<mailto:[e-mail]>>
Subject: Re: Negative disaggregation feature guide

[External Email. Be cautious of content]

 my thoughts:

negative is seriously more complex to  implement and understand operationally and only needed on multi-plane fabrics, taht's why positive is KISS ;-) Also, negative always forces you to ring the top of the fabric which in single plane design is an unnecessary requirement.

I foresee vendors not implementing negative for simplicity, very small footprint fabrics.

the discussion about capability advertisement is interesting, we can just add one bit on the node capabilities element, make it optional and default true.

Of course an implementation taht understands that all involved nodes understand negative is free to use negative instead of positive then.

-- tony

On Thu, May 14, 2020 at 11:05 AM Bruno Rijsman <[e-mail]<mailto:[e-mail]>> wrote:
Broadening a discussion on negative disaggregation to the RIFT mailing list.

Now that we have implemented negative disaggregation in RIFT-Python, and now that we are getting some operational experience with it, the following is becoming more and more evident:

It seems to us (Bruno and Pascal, for now) that once negative disaggregation is implemented, there is really no need for positive disaggregation any more.

Every use case that can be solved by positive disaggregation can (as far as we can currently tell) also be solved by negative disaggregation.

Furthermore, negative disaggregation solves the problems in a far simpler and more elegant way: there are fewer advertising nodes and fewer advertised TIEs involved, and we don’t have the synchronization issue that positive disaggregation has that potentially causes incast problems.

Thus, at the very least, it makes sense to recommend that in a given fabric we use either only negative disaggregation only or positive disaggregation only, with negative disaggregation being the default.

If positive and negative disaggregation are enabled simultaneously in one and the same fabric, everything still works fine as far as we can see. But there are some “interesting” interactions that make things unnecessarily complex and potentially fragile.

If support for negative disaggregation is mandatory, and if our assessment that negative disaggregation can solve all use cases is correct, then we could go one bold step further and completely remove positive disaggregation from the specification.

Thoughts?

— Bruno


On May 14, 2020, at 10:48 AM, Pascal Thubert (pthubert) <[e-mail]<mailto:[e-mail]>> wrote:

Hello Bruno:

I like your negative-only default because it is probably the safest. Let me elaborate..

partitioning saves a lot more ports per ToF node than the ring cost so it does not matter.

If you enable negative, there’s no point in doing any positive at all, is there? So what do we try to save by combining? In your example, I guess that super 1 2 had to retract the positive disag and then do negative instead when the link to spine 1 1 fell. Was that not a bit complex and error prone?

Also I’m not clear how we can always decide that dynamically and I do not believe that we specified that. So I understand that it’s more like a use case thingy, like a configuration that would be adapted to the use case.

=> do not configure both at the same time use either positive or negative disag.

We do not try to do positive disag transitively. So if there is a need for transitive, you have to use negative.  In a multiplane case, there’s usually (as in your picture) one ToP per plane in each PoD, and a first link failure (say Spine 1 1 to leaf 1 1) can already cause a fallen leaf: leaf 1 1 is no more reachable within plane 1. This tells you that all leaves in the other PoDs must avoid that plane, and that’s a leaf decision; which tells you that you need to recurse transitively down. This indicates that negative must be activated in a highly partitioned (meaning low redundancy) multiplane.

Say we try to use positive disaggregation transitively in a ToF that is partitioned like in planes, you’d run a risk that none of the ToF nodes that can still reach the leaf can see (through south reflection) the ToF node that cannot so they do not know they need to disag. So you end up needing the same ring as for negative, the southern reflection become mostly useless and the benefits of positive are gone.

Partitionned ToF => negative

I trust that we can use positive in a highly redundant and symmetrical non-partitioned ToF. If no one can reach the leaf anymore (like it’s dead) we have no issue since it should not be disaggregated, just use the discards route at any ToF node. If you have enough redundancy and no partition,  there will always be nodes that can both reach a leaf and discover that a peer does not.

=> Positive is indicated for a very specific use case, ideally 2 levels and fully meshed. Maybe that’s enough deployments to justify the feature in the code.

So the question is really that blurry zone between highly redundant single plane and highly partitioned multiplane what should one do? I becomes a risk/benefit judgement… If negative is implemented and the ring is there, I’d use it, better safe than sorry.

Take care

Pascal



From: Bruno Rijsman <[e-mail]<mailto:[e-mail]>>
Sent: jeudi 14 mai 2020 17:36
To: Pascal Thubert (pthubert) <[e-mail]<mailto:[e-mail]>>
Cc: Melchior Aelmans <[e-mail]<mailto:[e-mail]>>; Christian Graf <[e-mail]<mailto:[e-mail]>>; Oliver Steudler <[e-mail]<mailto:[e-mail]>>; Olivier Vandezande <[e-mail]<mailto:[e-mail]>>; Tony Przygienda <[e-mail]<mailto:[e-mail]>>; Antoni Przygienda <prz=[e-mail]<mailto:prz=[e-mail]>>; [e-mail]<mailto:[e-mail]>; Jeff Tantsura <[e-mail]<mailto:[e-mail]>>; Jeffrey (Zhaohui) Zhang <[e-mail]<mailto:[e-mail]>>; Jeffrey (Zhaohui) Zhang <zzhang=[e-mail]<mailto:zzhang=40juniper.net@dmarc..ietf.org>>; Jeffrey Zhang <[e-mail]<mailto:[e-mail]>>
Subject: Re: Negative disaggregation feature guide


On May 14, 2020, at 8:53 AM, Pascal Thubert (pthubert) <[e-mail]<mailto:[e-mail]>> wrote:

This does not show because on your first breakage you use the positive disag. If you stick to the logic you used at the beginning, that is pick an example where positive applies but use negative, then on your first breakage you’d show that the negative does not need to go transitively to the leaf, because the spine nodes still have solutions. Is that a lot of work to change?

This is exactly what I was thinking as well.

Maybe I need to go through the current specification with a fine-toothed comb again, but my (possibly incorrect) understanding is that the current version of the specification is that we positive disaggregation is used (MUST be used? SHOULD be used?) in the “first failure” scenario.

Personally, I see no reason why we would not be able to use negative disaggregation instead of positive disaggregation in the first failure scenario as well.

In the context of the tutorial, this would have the advantage of clearly demonstrating the propagation logic more clearly.

But more importantly, it seems to me that once negative disaggregation is implemented, there is really no reason to use positive disaggregation anywhere. It would be “cleaner” (less signaling) and “better” (no incast) to simply use negative disaggregation everywhere.

So…. I was thinking to add a configuration knob to my code “positive-only”, “negative-only”, “positive-and-negative”, with “negative-only” being the default? (Seems like a waste to have coded up the positive disaggregation).

What does everyone think of that?

Should we go even further and simply remove positive disaggregation from the spec?

Should we have negative disaggregation as a capability in the capability negotiation? (Seems non-sensical, given it is something that the whole fabric needs to support or not.)

Should we take this to the RIFT mailing list?

— Bruno

_______________________________________________
RIFT mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/rift