[Rift] Possible chain reaction of LIE messages with multiple neighbors on LAN?

[Bruno Rijsman <[e-mail]>]

Hi Tony and list,

I noticed some “interesting" behavior while testing multiple RIFT neighbors on a LAN:

When you have multiple neihghbors on a LAN, you can end up in a situation where the nodes send an exploding storm of LIE messages. The LIE messages are sent in a tight loop (they are triggered by state transitions, not by time expiries) and the number of messages increases exponentially very quickly. Wireshark sees hundreds of packets per second, which continues until you shut the entire topology down.

- For a wireshark traffic graph see http://bit.ly/rift-multiple-neighbor-chain-reaction-wireshark-traffic-graph

- For a wireshark trace see http://bit.ly/rift-multiple-neighbor-chain-reaction-wireshark-trace

- For a visualization of the events see http://bit.ly/rift-multiple-neighbor-chain-reaction-visualization

- For the topology details see http://bit.ly/rift-multiple-neighbor-chain-reaction-topology

I *think* this behavior is not implementation-specific but rather a direct consequence of what is written in the specification.

(It could be that I missed something in the speciciation, in which case I would be happy to stand corrected and fix my implementation.)

Here is the topology:

# Topology: multi_neighbor
#
#  +-----------+    +-----------+    +-----------+
#  | node1     |    | node2     |    | node3     |
#  | (level 0) |    | (level 1) |    | (level 2) |
#  +-----------+    +-----------+    +-----------+
#        | if1            | if1            | if1
#        |                |                |
#  =============================================== LAN
#
#  Adjacency will not reach 3-way because RIFT only allows point-to-point LANs
#  Furthermore, this topology uses the default RIFT multicast addresses and ports

Here is what I think happens:

At boot time, each node sends out a LIE message before it has received any LIE messages from any of the other nodes (because of the SendLie state entry action for state OneWay).

(1) Each node receives two LIE messages, one from each of the other nodes

(2) When the first LIE message is received, it triggers a NewNeighbor event, which causes a transition to state TwoWay and which also pushes a SendLie event (see item #7 on page 81 of draft-rift-02). This immediately (no delay) multicasts a LIE message which is received by both other nodes.

(3) When the second LIE message is received, it triggers a MultipleNeighbors event, which causes the node to transition back from state TwoWay to state OneWay. Because of the state entry action for state OneWay, this causes another immediate (no delay) SendLie which sends out a LIE message which is received by both other nodes.

Thus, when one node (say node 1) sends a LIE, it causes the other nodes (node 2 and 3 in this case) to *immediately* send out *two* LIE messages each. This is a run-away chain reaction in a tight loop: 1 LIE message immediately leads to 2 LIE messages, which immediately leads to 4 LIE messages, etc. etc.

As you scroll down in the visualization, you can see the forest of blue lines (which represent messsages) thickening into a spagheti mess. Note the time-stamps in the 2nd column on the left side: it all happens in a very short amount of time.

It was evidently a conscious decision in the spec to send LIE packets not only periodically but also one certain state transitions (e.g. as a state entry action for the ONE_WAY state). While this does indeed speed up (re)convergence you have to be very careful that there are no scenarios where you end up in an uncontrolled storm of LIE messages as this example demonstrates.

— Bruno