Re: [Rift] [RIFT] spine-tof links break => spine-leaf adjacencies go down?

[Bruno Rijsman <[e-mail]>]

Gotcha.

Thanks for confirming.

PS: It turned out that the issue we were seeing was caused by the fact that Travis CI/CD *still* does not support IPv6 in this day and age.

> On Apr 23, 2020, at 3:27 PM, Tony Przygienda <[e-mail]> wrote:
> 
> Absolutely intendend and expected behavior ;-) You can't figure out topology from spine down basically without a ToF. ToFs are the only source of truth where north is and if that isn't around no'one knows which way is up ... 
> 
> It's actually easy to put some code in that watches for the condition on a node & complains that there are no valid offers. 
> 
> Alternative would be to "store" offers and use them even if a node went away. Think through that, it's impossible to converge the protocol correctly then without having something like flooding of offers with seqnr# and so on ... 
> 
> --- tony 
> 
> On Thu, Apr 23, 2020 at 11:13 AM Bruno Rijsman <[e-mail] <mailto:[e-mail]>> wrote:
> While testing negative disaggregation with Mariano, I discovered an "interesting" behavior in RIFT and I would like to check whether this behavior is expected or not. I am not saying the behavior is wrong, I was just surprised to see it and wanted to check whether this was intended when the protocol was designed.
> 
> Consider the following topology:
> 
>  TOF1     TOF2
>    \       /
>     \     /
>      \   /
>      SPINE
>      /   \
>     /     \
>    /       \
>  LEAF1  LEAF2
> 
> All adjacencies are 3way.
> 
> The TOFs have top-of-fabric level 24.
> 
> The leaves have level 0.
> 
> The spine uses ZTP to derive a level, so it ends up with level 23.
> 
> Now we break all links between the spine and its TOFs:
> 
>  TOF1     TOF2
>    \       /
>     X     X   <<< These adj go down (as expected)
>      \   /
>      SPINE
>      /   \
>     /     \   <<< Surprise: these adj also go down
>    /       \
>  LEAF1  LEAF2
> 
> At this point the spine node does not have any Valid Offered Levels (VOLs) because (see [1]):
> (1) The levels from the TOFs are not valid because the adjacencies are down
> (2) The levels from leaves are never accepted as valid offers
> 
> Hence, the derived level for the spine becomes "undefined".
> 
> Once the level of the spine becomes undefined, it stops accepting LIE messages from the leaves because a node whose level is undefined never accepts LIEs (see [2] below).
> 
> Since the spine stops accepting LIEs from the leaves, the old LIEs time out and the adjacencies go down.
> 
> My question is: it is normal and expected that breaking the links from the spine to the TOFs *also* causes the adjacencies between th spine and the leaves to go down?
> 
> [1] On page [86] of draft-11 it says:
> 
>  Valid Offered Level (VOL): A neighbor’s level received on a valid
>  LIE (i.e. passing all checks for adjacency formation while
>  disregarding all clauses involving level values) persisting for
>  the duration of the holdtime interval on the LIE. Observe that
>  offers from nodes offering level value of 0 do not constitute VOLs
>  (since no valid DERIVED_LEVEL can be obtained from those and
>  consequently ‘not_a_ztp_offer‘ MUST be ignored). Offers from LIEs
>  with ‘not_a_ztp_offer‘ being true are not VOLs either. If a node
>  maintains parallel adjacencies to the neighbor, VOL on each
>  adjacency is considered as equivalent, i.e. the newest VOL from
>  any such adjacency updates the VOL received from the same node.
> 
> [2] On page [45] of draft -11 it says:
> 
>  4. PROCESS_LIE:
>  1. if lie has wrong major version OR our own system ID or
>  invalid system ID then CLEANUP else
>  2. if lie has non matching MTUs then CLEANUP, PUSH
>  UpdateZTPOffer, PUSH MTUMismatch else
>  3. if PoD rules do not allow adjacency forming then CLEANUP,
>  PUSH PODMismatch, PUSH MTUMismatch else
>  4. if lie has undefined level OR my level is undefined OR this
>  node is leaf and remote level lower than HAT OR (lie’s level
>  is not leaf AND its difference is more than one from my
>  level) then CLEANUP, PUSH UpdateZTPOffer, PUSH
>  UnacceptableHeader else
>