Re: [Rift] AD Review of draft-ietf-rift-rift-12 (Part 1)

["Pascal Thubert (pthubert)" <[e-mail]>]

Hello Alvaro:

As part of the weekly RIFT meet&progress; I'm looking at the nits in 4.1.2 to 4.1.4:



> 
> 705	   To account for the "northern" and the "southern" information split
> 706	   the link state database is partitioned accordingly into "north
> 707	   representation" and "south representation" TIEs.  In simplest terms
> 708	   the North TIEs contain a link state topology description of lower
> 709	   levels and and South TIEs carry simply default routes towards the
> 710	   level above.  This oversimplified view will be refined gradually in
> 711	   following sections while introducing protocol procedures and state
> 712	   machines at the same time.
> 
> [nit] s/in following/in the following

Done

> 
> 
> 714	4.1.2.  Generalized Topology View
> 
> 716	   This section will shed some light on the topologies RIFT addresses,
> 717	   including multi plane fabrics and their implications.  Readers that
> 718	   are only interested in single plane designs, i.e. all top-of-fabric
> 719	   nodes being topologically equal and initially connected to all the
> 720	   switches at the level below them, can skip the rest of Section 4.1.2
> 721	   and resulting Section 4.2.5.2 as well.
> 
> [minor] "Readers...can skip the rest of Section 4.1.2 and resulting Section
> 4.2.5.2 as well."  I can see how a reader can skip a part of the overview, but
> §4.2* is where the specification is.  Are you saying that §4.2.5.2 doesn't have to
> be implemented/supported in some cases?
> Are there other sections that are also not needed in some cases?  Does this
> result in the ability to implement subsets of RIFT to support specific
> topologies?  Where is that discussed?

As you know there's more is in the applicability draft, and we probably do not want to diverge too much in this spec.

To address both points above I suggest we change to
"
   [I-D.ietf-rift-applicability] discusses various topologies upon which
   RIFT may be deployed and use cases where it can apply.  This section
   and resulting section 3.2.5.2 are dedicated to multi-plane fabrics,
   in contrast with the single plane designs where all top-of-fabric
   nodes are topologically equal and initially connected to all the
   switches at the level below them.
"

 
> 
> ...
> 737	4.1.2.1.  Terminology
> ...
> 746	   K: Denotes the number of ports in radix of a switch pointing north or
> 747	      south.  Further, K_LEAF denotes number of ports pointing south,
> 748	      i.e. towards leaves, and K_TOP for number of ports pointing north
> 749	      towards a higher spine level.  To simplify the visual aids,
> 750	      notations and further considerations, K will be mostly set to
> 751	      Radix/2.
> 
> [minor] Radix is defined in §3.1 as the number of ports.  s/Denotes the number
> of ports in radix of a switch/Denotes the radix of a switch
> 
Done

> 
> ...
> 757	   N: Denote the number of independent ToF planes in a topology.
> 
> [nit] s/Denote/Denotes
> 

Done
	
> 
> ...
> 766	4.1.2.2.  Clos as Crossed Crossbars
> 
> 768	   The typical topology for which RIFT is defined is built of P number
> 769	   of PoDs and connected together by S number of ToF nodes.  A PoD
> node
> 770	   has K number of ports (also called Radix).  We consider half of them
> 771	   (K=Radix/2) as connecting host devices from the south, and the other
> 772	   half connecting to interleaved PoD Top-Level switches to the north.
> 773	   Ratio K can be chosen differently without loss of generality when
> 774	   port speeds differ or the fabric is oversubscribed but K=R/2 allows
> 775	   for more readable representation whereby there are as many ports
> 776	   facing north as south on any intermediate node.  We represent a
> node
> 777	   hence in a schematic fashion with ports "sticking out" to its north
> 778	   and south rather than by the usual real-world front faceplate designs
> 779	   of the day.
> 
> [nit] s/Ratio K can be chosen differently/The K ratio can be chosen differently
> 
OK
> 
> [minor] "K=R/2"  R is defined in §4.1.2.1 as the redundancy, not the radix.

Yes that created confusion. Changed to K=Radix/2

> 
> 
> 781	   Figure 4 provides a view of a leaf node as seen from the north, i.e.
> 782	   showing ports that connect northbound.  For lack of a better symbol,
> 783	   we have chosen to use the "o" as ASCII visualisation of a single
> 784	   port.  In this example, K_LEAF has 6 ports.  Observe that the number
> 785	   of PoDs is not related to Radix unless the ToF Nodes are constrained
> 786	   to be the same as the PoD nodes in a particular deployment.
> 
> [minor] "showing ports that connect northbound...K_LEAF has 6 ports"
> The ports that connect north are K_TOP.

K is a number. The Radix discussion is about have all the ports the same speed, half of them north.
From the perspective of a leaf that is K_LEAF ports north. For the ToP that is K_TOP ports south.
Everything wires if K_TOP * nbToP = K_LEAF * nbLeaf


> 
> 
> 788	       Top view
> 789	        +---+
> 790	        |   |
> 791	        | o |     e.g., Radix = 12, K_LEAF = 6
> 792	        |   |
> 793	        | o |
> 794	        |   |      -------------------------
> 795	        | o ------- Physical Port (Ethernet) ----+
> 796	        |   |      -------------------------     |
> 797	        | o |                                    |
> 798	        |   |                                    |
> 799	        | o |                                    |
> 800	        |   |                                    |
> 801	        | o |                                    |
> 802	        |   |                                    |
> 803	        +---+                                    |
> 
> 805	          ||             ||      ||      ||      ||      ||      ||
> 806	        +----+       +------------------------------------------------+
> 807	        |    |       |                                                |
> 808	        +----+       +------------------------------------------------+
> 809	          ||             ||      ||      ||      ||      ||      ||
> 810	              Side views
> 
> 812	                      Figure 4: A Leaf Node, K_LEAF=6
> 
> 814	   The Radix of a PoD's top node may be different than that of the leaf
> 815	   node.  Though, more often than not, a same type of node is used for
> 816	   both, effectively forming a square (K*K).  In general case, we could
> 817	   have switches with K_TOP southern ports on nodes at the top of the
> 818	   PoD which are not necessarily the same as K_LEAF.  For instance, in
> 819	   the representations below, we pick a 6 port K_LEAF and a 8 port
> 820	   K_TOP.  In order to form a crossbar, we need K_TOP Leaf Nodes as
> 821	   illustrated in Figure 5.
> 
> [nit] s/In general case/In the general case
> 
Done

> 
> [minor] "K_TOP southern ports"  Aren't K_TOP the ports pointing north?

No they K_TOP is ToP's Radix divided by to, that's the number of ports the ToP has going south and north

>  The description is confusing because the terminology from the last section is
> not used in the same way -- the description mixes the terminology with the
> number represented.  For example, "K_TOP Leaf Nodes" doesn't make sense if
> the terminology is strictly applied, where K_TOP is the "number of ports
> pointing north".  Also (if I understood Figure 4 correctly), each node below has
> 6 K_TOP ports -- presumably the node at the top has 8 K_LEAF ports.
> 
> 
> 823	             +---+  +---+  +---+  +---+  +---+  +---+  +---+  +---+
> 824	             |   |  |   |  |   |  |   |  |   |  |   |  |   |  |   |
> 825	             | o |  | o |  | o |  | o |  | o |  | o |  | o |  | o |
> 826	             |   |  |   |  |   |  |   |  |   |  |   |  |   |  |   |
> 827	             | o |  | o |  | o |  | o |  | o |  | o |  | o |  | o |
> 828	             |   |  |   |  |   |  |   |  |   |  |   |  |   |  |   |
> 829	             | o |  | o |  | o |  | o |  | o |  | o |  | o |  | o |
> 830	             |   |  |   |  |   |  |   |  |   |  |   |  |   |  |   |
> 831	             | o |  | o |  | o |  | o |  | o |  | o |  | o |  | o |
> 832	             |   |  |   |  |   |  |   |  |   |  |   |  |   |  |   |
> 833	             | o |  | o |  | o |  | o |  | o |  | o |  | o |  | o |
> 834	             |   |  |   |  |   |  |   |  |   |  |   |  |   |  |   |
> 835	             | o |  | o |  | o |  | o |  | o |  | o |  | o |  | o |
> 836	             |   |  |   |  |   |  |   |  |   |  |   |  |   |  |   |
> 837	             +---+  +---+  +---+  +---+  +---+  +---+  +---+  +---+
> 
> 839	                 Figure 5: Southern View of a PoD, K_TOP=8
> 
> 841	   As further visualized in Figure 6 the K_TOP Leaf Nodes are fully
> 842	   interconnected with the K_LEAF PoD-top nodes, providing
> connectivity
> 843	   that can be represented as a crossbar when "looked at" from the
> 844	   north.  The result is that, in the absence of a failure, a packet
> 845	   entering the PoD from the north on any port can be routed to any
> port
> 846	   in the south of the PoD and vice versa.  And that is precisely why it
> 847	   makes sense to talk about a "switching matrix".
> 
> [minor] "K_TOP Leaf Nodes are fully interconnected with the K_LEAF PoD-top
> nodes"  Same comment about the terminology...   I only see one "PoD top
> Node" with one connection to a switch, not a full interconnect.

I replaced the legacy terminology PoD-top to ToP (Top of PoD). In Fig 5 you look from southn that's the perspective of the servers in the PoD. The see the access ports in the the ToR == Leaf. The are K-ToP leaf switches that's 8 in this example, each switch as 6 access ports (Radix/2) since half of the ports look north and half look south


> 
> 
> [minor] The figure also doesn't show the connection between the switches (if
> any)...and I'm not sure what the "connectors" (?) on the switch at the
> top/bottom are (there seem to be more of them than ports).

There is no connection between the ToRs. On the southern side that's just a bunch of ports. They will be interconnected on the northern side by the ToPs and that's fig 6 below:
> 
> 
> 849	                                      E<-*->W
> 
> 851	          +---+  +---+  +---+  +---+  +---+  +---+  +---+  +---+
> 852	          |   |  |   |  |   |  |   |  |   |  |   |  |   |  |   |
> 853	        +--------------------------------------------------------+
> 854	        |   o      o      o      o      o      o      o      o   |
> 855	        +--------------------------------------------------------+
> 856	        +--------------------------------------------------------+
> 857	        |   o      o      o      o      o      o      o      o   |
> 858	        +--------------------------------------------------------+
> 859	        +--------------------------------------------------------+
> 860	        |   o      o      o      o      o      o      o      o   |
> 861	        +--------------------------------------------------------+
> 862	        +--------------------------------------------------------+
> 863	        |   o      o      o      o      o      o      o      o   |
> 864	        +--------------------------------------------------------+
> 865	        +--------------------------------------------------------+
> 866	        |   o      o      o      o      o      o      o      o   |<-+
> 867	        +--------------------------------------------------------+  |
> 868	        +--------------------------------------------------------+  |
> 869	        |   o      o      o      o      o      o      o      o   |  |
> 870	        +--------------------------------------------------------+  |
> 871	          |   |  |   |  |   |  |   |  |   |  |   |  |   |  |   |    |
> 872	          +---+  +---+  +---+  +---+  +---+  +---+  +---+  +---+    |
> 873	                     ^                                              |
> 874	                     |                                              |
> 875	                     |     ----------        ---------------------  |
> 876	                     +----- Leaf Node        PoD top Node (Spine) --+
> 877	                           ----------        ---------------------
> 
> 879	            Figure 6: Northern View of a PoD's Spines, K_TOP=8
> 
> 881	   Side views of this PoD is illustrated in Figure 7 and Figure 8.
> 
> 883	                      Connecting to Spine
> 
> 885	      ||      ||      ||      ||      ||      ||      ||      ||
> 886	  +----------------------------------------------------------------+   N
> 887	  |                    PoD top Node seen sideways                  |   ^
> 888	  +----------------------------------------------------------------+   |
> 889	      ||      ||      ||      ||      ||      ||      ||      ||       *
> 890	    +----+  +----+  +----+  +----+  +----+  +----+  +----+  +----+     |
> 891	    |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |     v
> 892	    +----+  +----+  +----+  +----+  +----+  +----+  +----+  +----+     S
> 893	      ||      ||      ||      ||      ||      ||      ||      ||
> 
> 895	                           Connecting to Client nodes
> 
> 897	              Figure 7: Side View of a PoD, K_TOP=8, K_LEAF=6
> 
> [minor] I count 8 connections to the south in the top node...and just one on the
> switches below it.

A leaf switch is represented as a vertical bar, 8 of them. A ToP is represented as a horizontal, 6 of them.
The   o  in the cross-bar representation symbolizes the ethernet cable that connects them. 


> 
> 
> 899	                   Connecting to Spine
> 
> 901	          ||      ||      ||      ||      ||      ||
> 902	        +----+  +----+  +----+  +----+  +----+  +----+              N
> 903	        |    |  |    |  |    |  |    |  |    |  |   PoD top Nodes   ^
> 904	        +----+  +----+  +----+  +----+  +----+  +----+              |
> 905	          ||      ||      ||      ||      ||      ||                *
> 906	      +------------------------------------------------+            |
> 907	      |              Leaf seen sideways                |            v
> 908	      +------------------------------------------------+            S
> 909	          ||      ||      ||      ||      ||      ||
> 
> 911	                   Connecting to Client nodes
> 
> [minor] A leaf doesn't have southbound ports/adjacencies.  What is this leaf
> connected to?
> 
> 
> 913	    Figure 8: Other Side View of a PoD, K_TOP=8, K_LEAF=6, 90o turn in
> 914	                                 E-W Plane
> 
> [minor] In this case I count a leaf with 6 northbound interfaces.
> 
> 
> [minor] "90o turn in E-W Plane"  I don't know what that is.
> 
> 
> 916	   As next step, let us observe that a resulting PoD can be abstracted
> 917	   as a bigger node with a number K of K_POD= K_TOP * K_LEAF, and
> the
> 918	   design can recurse.
> 
> [minor] K is already defined as the number of ports (§4.1.2.1).
> 
> 
> [minor] Lost again.  If the PoD is abstracted as a single node, then it would have
> K_TOP + K_LEAF nodes, not sure where the "*" comes from or what is trying to
> denote.
> 
> 
> 920	   It will be critical at this point that, before progressing further,
> 921	   the concept and the picture of "crossed crossbars" is clear.  Else,
> 922	   the following considerations might be difficult to comprehend.
> 
> [] The concept is clear to me -- I don't find the explanation and the
> corresponding pictures specially helpful.
> 
> 
> ...
> 929	   This topology is also referred to as a single plane configuration and
> 930	   is quite popular due to its simplicity.  In order to reach a 1:1
> 931	   connectivity ratio between the ToF and the leaves, it results that
> 932	   there are K_TOP ToF nodes, because each port of a ToP node
> connects
> 933	   to a different ToF node, and K_LEAF ToP nodes for the same reason.
> 934	   Consequently, it will take (P * K_LEAF) ports on a ToF node to
> 935	   connect to each of the K_LEAF ToP nodes of the P PoDs, as shown in
> 936	   Figure 9.
> 
> [minor] "there are K_TOP ToF nodes...and K_LEAF ToP nodes"  As with other
> places, the terminology is not used as defined earlier.  K_* refer to the number
> of ports in a specific switch, so their use is relative to that switch.  In this case
> each ToP has K_TOP links, and each ToF has K_LEAF links.  The use without the
> reference point is confusing.
> 
> 
> [minor] "(P * K_LEAF)"  This calculation is clear once one realizes that the
> previous discussion was for the number of ports per PoD, not total (as the
> definition of K_* suggests).
> 
> 
> 
> 938	        [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] <-----+
> 939	         |   |   |   |   |   |   |   |        |
> 940	      [=================================]     |     -----------
> 941	         |   |   |   |   |   |   |   |        +----- Top-of-Fabric
> 942	        [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]       +----- Node      -------+
> 943	                                              |     -----------       |
> 944	                                              |                       v
> 945	        +-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+ <-----+                      +-+
> 946	        | | | | | | | | | | | | | | | |                              | |
> 947	      [ |o| |o| |o| |o| |o| |o| |o| |o| ]                            | |
> 948	      [ |o| |o| |o| |o| |o| |o| |o| |o| ] -------------------------  | |
> 949	      [ |o| |o| |o| |o| |o| |o| |o| |o<--- Physical Port (Ethernet)  | |
> 950	      [ |o| |o| |o| |o| |o| |o| |o| |o| ] -------------------------  | |
> 951	      [ |o| |o| |o| |o| |o| |o| |o| |o| ]                            | |
> 952	      [ |o| |o| |o| |o| |o| |o| |o| |o| ]                            | |
> 953	        | | | | | | | | | | | | | | | |                              | |
> 954	      [ |o| |o| |o| |o| |o| |o| |o| |o| ]                            | |
> 955	      [ |o| |o| |o| |o| |o| |o| |o| |o| ]      --------------        | |
> 956	      [ |o| |o| |o| |o| |o| |o| |o| |o| ] <---  PoD top level        | |
> 957	      [ |o| |o| |o| |o| |o| |o| |o| |o| ]       node (Spine)  ---+   | |
> 958	      [ |o| |o| |o| |o| |o| |o| |o| |o| ]      --------------    |   | |
> 959	      [ |o| |o| |o| |o| |o| |o| |o| |o| ]                        |   | |
> 960	        | | | | | | | | | | | | | | | |  -+           +-   +-+   v   | |
> 961	      [ |o| |o| |o| |o| |o| |o| |o| |o| ] |           |  --| |--[ ]--| |
> 962	      [ |o| |o| |o| |o| |o| |o| |o| |o| ] |   -----   |  --| |--[ ]--| |
> 963	      [ |o| |o| |o| |o| |o| |o| |o| |o| ] +--- PoD ---+  --| |--[ ]--| |
> 964	      [ |o| |o| |o| |o| |o| |o| |o| |o| ] |   -----   |  --| |--[ ]--| |
> 965	      [ |o| |o| |o| |o| |o| |o| |o| |o| ] |           |  --| |--[ ]--| |
> 966	      [ |o| |o| |o| |o| |o| |o| |o| |o| ] |           |  --| |--[ ]--| |
> 967	        | | | | | | | | | | | | | | | |  -+           +-   +-+       | |
> 968	        +-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+                              +-+
> 
> 970	      Figure 9: Fabric Spines and TOFs in Single Plane Design, 3 PoDs
> 
> [minor] I believe you when you say that this figure shows how "it will take (P *
> K_LEAF) ports on a ToF node to connect to each of the K_LEAF ToP nodes of the
> P PoDs", but the drawing is not straight forward to interpret.  Among other
> reasons because there seem to be 3 different connection
> types/interpretations/? to the ToF -- a different one for each PoD.

Each pod is a crossbar as before, like a pizza. 3 of net next to one another in a line.
Then on top of the 3 pizzas you place long bars parallel to the leaves that span the 3 pizzas to interconnect them. These are the ToF. 
The view above is top (the large image on bottom left and 2 sides, rotated 90 degrees.
It takes a little bit to see it but you get used.

To clarify I added 

"
                                                  Figure 9
   illustrates this, looking at P=3 PoDs from above and the 2 sides.  The
   large view is the one from above, with the 8 ToF of 3*6 ports each
   interconnecting the PoDs, every ToP Node being connected to every ToF
   node.
"
> 
> 
> 972	   The top view can be collapsed into a third dimension where the
> hidden
> 973	   depth index is representing the PoD number.  We can then show one
> PoD
> 974	   as a class of PoDs and hence save one dimension in our
> 975	   representation.  The Spine Node expands in the depth and the vertical
> 976	   dimensions, whereas the PoD top level Nodes are constrained, in
> 977	   horizontal dimension.  A port in the 2-D representation represents
> 978	   effectively the class of all the ports at the same position in all
> 979	   the PoDs that are projected in its position along the depth axis.
> 980	   This is shown in Figure 10.
> 
> [] Do we really need this extra representation?

Yes because that helps the reader to make the required additional step to project all Pods as one and work at the class level.
IOW, one pod now is the class of all PoDs, it represents them all, all PoDs stacked and seen from above as one.
After that drawings can simplify a lot because we omit the PoD dimension and can draw in a 2-D ASCII world.

> 
> ...
> 1003	   As simple as single plane deployment is it introduces a limit due to
> 1004	   the bound on the available radix of the ToF nodes that has to be at
> 1005	   least P * K_LEAF.  Nevertheless, we will see that a distinct
> 1006	   advantage of a connected or non-partitioned Top-of-Fabric is that all
> 1007	   failures can be resolved by simple, non-transitive, positive
> 1008	   disaggregation (i.e. nodes advertising more specific prefixes with
> 1009	   the default to the level below them that is however not propagated
> 1010	   further down the fabric) as described in Section 4.2.5.1 . In other
> 1011	   words; non-partitioned ToF nodes can always reach nodes below or
> 1012	   withdraw the routes from PoDs they cannot reach
> unambiguously.  And
> 1013	   with this, positive disaggregation can heal all failures and still
> 1014	   allow all the ToF nodes to see each other via south reflection





Sorry this appears to be cut here ?

Pascal