Re: [Rift] RIFT strange behaviours discussed today

[Bruno Rijsman <[e-mail]>]

Hi Mariano and Tomasso,


I have made two big changes to the way messages are queued in RIFT-Python. 

This code has been committed to the master branch.

So it is ready for you to test, just in time for the paper deadline. Although it is a bit risky to introduce such a large change with not much time to fully “soak it in” before the paper deadline. Also, starting Monday morning I will be unreachable for a few days.  I leave it up to you to decide whether or not you want to use this new code for the paper.


Big change #1

In the old code, the TIES_TX, TIES_ACK, and TIES_REQ queue were all serviced only once per second, no matter what. The TIES_RTX queue didn’t really do anything.

TIDEs were sent once per 2 seconds (that did not change).

In the new code, there are two queue for TIEs, TIE ACKs, and TIE REQs:

(a) A fast tx_queue for the initial transmission after the item is enqueued. It is serviced every 50ms. (With an optimization to make sure we don’t run such a fast timer unless it is really needed - i.e. unless there is at least one entry in some fast queue.)

(b) A slow rtf_queue for subsequent retransmission if needed. It is serviced every 1000ms.

See the new file msg_queues.py for details.


Big change #2

In the old code, if a node regenerated a local TIE for any reason, it was not immediately reflooded.

Instead we would have to wait up to 2 seconds for the regenerated TIE to be advertised in a TIDE, and then up to 1 second for the other side to request it, and the up to another 1 second for this node to react to the request.

I changed that: if a local TIE is regenerated for any reason, the TIE itself is immediately put on the fast TIE tx_queue, so it will be sent after 50ms.


These two changes combined make most of the problems that you saw go away.

You used see that a node would send a TIE, and then a second later the exact same version of that TIE (same tie-nr, same seq-nr).

This was because the TIRE was not sent fast enough to ack the TIE.

That behavior should go away — the TIRE is now sent much faster.

In general, convergence should be much faster.


You may see some other behavior that you did not see before.

If you kill a node in the topology, you may see some “additional intermediate states” because reconvergence is so much faster.

Consider for example, that node X is killed, and node X had adjacencies with Y1, Y2, Y3, Y4, …., Yn

Each of those neighbor nodes Y1, Y2, …., Yn will lose their adjacency with X, and reflood their local node TIE.

Now, consider that Y1, Y2, …., Yn are also all adjacent to node Z (this is quite common in Clos topologies).

So, Z is going to receive updated node TIEs from Y1, Y2, Y3, …., Yn.

And (this is the important part), Z may ALSO receive pos-disagg-prefix and neg-disagg-prefix TIEs from Y1, Y2, …., Yn.

Z receives these multiple pos-disagg-prefix and neg-disagg-prefix messages asynchronously.

That may cause Z to “change it mind” a few times about whether and what it should disaggregate itself, and that may cause Z to originate multiple different versions (i.e. sequence numbers) of its own pos-disagg-prefix or neg-disagg-prefix message in quick succession.

This will also have the effect to sending more messages than you might expect, but this is really different from the behavior you were seeing before.

Before you were seeing the identical TIE (same seq-nr) being sent multiple times.  I would consider that to be real bug.

Now you might see multiple versions of the same TIE (same tie-nr, different seq-nr) being sent multiple times.  I don’t consider that to be a bug.  Not a bug in the code at least.  It is just a consequence of the RIFT protocol reacting quickly to multiple adjacencies going down in quick succession after a node failure.


I have a gut feeling that this “hunting” behavior will be less if we do negative disaggregation everywhere (and no positive disaggregation).

I am adding a “disaggregation: negative-only” configuration knob to test that hypothesis.

Don’t try it out yet, the code for this new knob is not finished yet. 

I will update you when it is.


— Bruno








> On May 15, 2020, at 10:43 AM, Mariano Scazzariello <[e-mail]> wrote:
> 
> Hi Bruno,
> any news with the duplicated packets issue?
> 
> We just want to know that since 19th of May is the abstract submission deadline, so we should know if we can match it in time.
> Our "internal deadline" is Sunday 17th of May.
> 
> Thanks,
> Mariano and Tommaso.
> 
> Il 12/05/2020 13:14, Mariano Scazzariello ha scritto:
>> Hi Bruno,
>> sorry for the spamming :D.
>> 
>> Today me and Tommy further investigated on the problem that I reported yesterday. 
>> It seems that it is caused by TIEs sent multiple times by the same node.
>> We are not sure, we'll keep investigating, however we would like to have a feedback from you.
>> 
>> 
>> EXAMPLE OF MULTIPLE NEG DISAGG TIES
>> This example highlights what we saw yesterday during the call.
>> The first two packets are correct, since the ToFs send them to spine_2_1_1.  However after some time, the same ToFs resend the same neg disagg TIEs, in fact the comparison is 0 (in red).
>> 
>> Sender                    Receiver                 Originator     TIE Type                    In DB? If so, comparison result.
>> tof_1_2_1:if1         spine_2_1_1         121                 Neg-Dis-Prefix          No
>> tof_1_2_2:if1         spine_2_1_1         122                 Neg-Dis-Prefix          No
>> ....
>> tof_1_2_1:if1         spine_2_1_1         121                 Neg-Dis-Prefix         Yes, comparison is 0
>> tof_1_2_2:if1         spine_2_1_1         122                 Neg-Dis-Prefix         Yes, comparison is 0
>> ...
>> tof_1_2_2:if1         spine_2_1_1         122                 Neg-Dis-Prefix         Yes, comparison is 0
>> 
>> EXAMPLE OF MULTIPLE NODE TIES
>> Here we can see that the ToFs send to spine_2_1_1 a new node TIE that is stored and reflected correctly (in green). Strange TIEs are highlighted in red.
>> 
>> Sender                     Receiver                    Originator     TIE Type                   In DB? If so, comparison result.
>> tof_1_2_1:if1         spine_2_1_1             121                 Node                         Yes, comparison is -1
>> tof_1_2_2:if1         spine_2_1_1             122                 Node                         Yes, comparison is -1
>> spine_2_1_1:if2     tof_1_2_1                 122                 Node                         Yes, comparison is -1                        <- CORRECT REFLECTION
>> spine_2_1_1:if3     tof_1_2_2                 121                 Node                         Yes, comparison is -1                        <- CORRECT REFLECTION
>> ...
>> tof_1_2_1:if1         spine_2_1_1             121                 Node                         Yes, comparison is -1                       <- Spine receives Node TIE from tof_1_2_1 different from the one stored in the DB, should reflect to tof_1_2_2
>> tof_1_2_2:if1         spine_2_1_1             122                 Node                         Yes, comparison is -1                       <- Spine receives Node TIE from tof_1_2_2 different from the one stored in the DB, should reflect to tof_1_2_1
>> spine_2_1_1:if2     tof_1_2_1                 122                 Node                         Yes, comparison is 0                        <- Reflection of tof_1_2_2 TIE to tof_1_2_1. Why comparison is 0 on ToF, if it is -1 on spine (it reflects wrong TIE?)?
>> spine_2_1_1:if3     tof_1_2_2                 121                 Node                         Yes, comparison is 0                        <- Reflection of tof_1_2_1 TIE to tof_1_2_2. Why comparison is 0 on ToF, if it is -1 on spine (it reflects wrong TIE?)?
>> ...
>> tof_1_2_1:if1         spine_2_1_1             121                 Node                         Yes, comparison is 0                        <- tof_1_2_1 resends to spine_2_1_1 its own node TIE
>> tof_1_2_2:if1         spine_2_1_1             122                 Node                         Yes, comparison is 0                        <- tof_1_2_2 resends to spine_2_1_1 its own node TIE
>> 
>> Hope this is useful to hunt the problem!
>> Mariano and Tommaso.
>> 
>> Il 12/05/2020 00:55, Mariano Scazzariello ha scritto:
>>> Another little update, the last for today since it's 1AM :D
>>> 
>>> The final scenario is:
>>> tof_1_2_2 sends its node TIE to its southbound adjacencies (spine_2_1_1/spine_3_1_1/spine_4_1_1). Each spine reflects it to tof_1_2_1.
>>> tof_1_2_1 resends this TIE to spine_2_1_1/spine_3_1_1/spine_4_1_1. Spines bounce it back to tof_1_2_2.
>>> 
>>> So, the final question is: is that correct that the ToF re-sends southbound the node TIE received by southern reflection? Or is this a strange behaviour?
>>> 
>>> Good night,
>>> Mariano and Tommaso.
>>> 
>>> Il 12/05/2020 00:06, Mariano Scazzariello ha scritto:
>>>> Little update:
>>>> the southern reflection is working properly. What we're seeing is this behaviour (similar to n.3 of previous mail). As example:
>>>> 
>>>> After spine_1_1_1 failure, tof_1_2_1 sends a node TIE (with seq n.7, originated by tof_1_2_2) to spine_2_1_1/spine_3_1_1/spine_4_1_1. Spines bounce it back to tof_1_2_2.
>>>> 
>>>> It seems that tof_1_2_1 reflects something that is already coming from a reflection since originator is tof_1_2_2 (?) Is that possible? Also, is this correct that a TIE is reflected to the same node that originated it?
>>>> 
>>>> We'll keep you updated.
>>>> Mariano and Tommaso.
>>>> 
>>>> Il 11/05/2020 23:09, Mariano Scazzariello ha scritto:
>>>>> Hi Bruno,
>>>>> as disussed today we'll report the strange behaviours found in RIFT.
>>>>> 
>>>>> 1. ToFs/Spines sending more than one neg disagg TIE
>>>>> Steps to reproduce: build a FT with K=2 and R=1, after convergence destroy spine_1_1_1. This is the state after the failure (don't mind the numbers :D).
>>>>> <jdcclkbcdlgpfggo.png>
>>>>> 
>>>>> In the figure below, spine_3_1_1 interface 0 (connected to leaf_3_0_1) sends 2 neg disagg TIEs after failure time.
>>>>> 
>>>>> <jnilpccgldahgdmk.png>
>>>>> 
>>>>> 2. Southern Reflection bounces PDUs back to the sender
>>>>> 
>>>>> Same scenario as before, spine_4_1_1 interface 3 (connected to tof_1_2_2) bounces back packets to tof_1_2_2 instead of sending them to tof_1_2_1
>>>>> <fgffmbbocoomlknk.png>
>>>>> UPDATE: We found that, for some reason, spine_3_1_1 sends reflection correctly. Interface 3 (connected to tof_1_2_2) sends TIEs coming from tof_1_2_1:
>>>>> <hbodnfifioifnhng.png>
>>>>> Me and Tommaso are investigating on that, however we have some difficulties on finding the code that reflects TIEs. Is the `unsol_flood_tie_packet_info` in Node class, right?
>>>>> 
>>>>> 3. Another strange behaviour?
>>>>> 
>>>>> tof_1_2_2 sends to spine_2_1_1 some Node TIEs originated by tof_1_2_1, why?
>>>>> <poionlajflebeigb.png>
>>>>> This screen wraps both the duplicated packets problem (229-233) and packet 228 is the "strange one".
>>>>> 
>>>>> Hope to hear from you soon,
>>>>> Mariano and Tommaso.
>>>>>