Re: AD review of draft-ietf-rtgwg-bgp-pic-18

[Alvaro Retana <[e-mail]>]

Ping!

On September 8, 2022 at 11:46:05 AM, Alvaro Retana ([e-mail])
wrote:

Ping!

On August 2, 2022 at 4:23:53 PM, Alvaro Retana ([e-mail])
wrote:


Dear authors:

Thank you for documenting this important feature.

After reading the document, I think that it still needs work:

(1) The terminology used is not aligned with rfc4271.  Of major importance
is
    the description of the Routing Table, the Decision Process, the use of
best
    routes (not paths!), etc.  I pointed out multiple occurrences below,
buy I
    need you to check the whole docuent for consistency.

(2) The choice to describe the mechanism using a complex scenario (VPNs,
etc.)
    has resulted in the text being more complex than it needs to be.

    It is probably too late for this, but a better approach (especially
because
    this is an Informational document) would have been to describe the
    functionality using a simple IP-only network.  A more elaborate example
    could have then been included in an Appendix.

(3) The text (mainly in §5) related to hardware limitations is interesting
but
    also highly speculative.  Given that it is not part of the main
mechanism,
    I would suggest putting this information in an appendix.


I put detailed comments below.


To the Chairs: I couldn't find a request to the idr WG for review.  Did I
miss it?  Given the subject, we need to give idr the opportunity to
comment.


Thanks!

Alvaro.


[Line numbers from idnits.]

...
12 Abstract

14 In a network comprising thousands of BGP peers exchanging millions of
15 routes, many routes are reachable via more than one next-hop. Given
16 the large scaling targets, it is desirable to restore traffic after
17 failure in a time period that does not depend on the number of BGP
18 prefixes. In this document we proposed an architecture by which
19 traffic can be re-routed to ECMP or pre-calculated backup paths in a
20 timeframe that does not depend on the number of BGP prefixes. The
21 objective is achieved through organizing the forwarding data
22 structures in a hierarchical manner and sharing forwarding elements
23 among the maximum possible number of routes. The proposed technique
24 achieves prefix independent convergence while ensuring incremental
25 deployment, complete automation, and zero management and provisioning
26 effort. It is noteworthy to mention that the benefits of BGP Prefix
27 Independent Convergence (BGP-PIC) are hinged on the existence of more
28 than one path whether as ECMP or primary-backup.

[style nit] "we proposed"

Please don't write using the first person ("we propose"), use the third
person instead ("this document proposes").


[minor] s/we proposed an architecture/describes an architecture

Other places also say that the document "proposes" -- at this point in the
process we're past proposals. :-)


[minor] Please expand ECMP in the Abstract, and later on first use.


[] "The proposed technique achieves prefix independent convergence while
ensuring incremental deployment, complete automation, and zero management
and provisioning effort."

Great marketing!  This is a technical document, please avoid selling the
solution.


[minor] Consider breaking the Abstract into two paragraphs.



...
109 1. Introduction

111   BGP speakers exchange reachability information about
112   prefixes[1][2] and, for labeled address families, namely AFI/SAFI
113   1/4, 2/4, 1/128, and 2/128, an edge router assigns local labels to
114   prefixes and associates the local label with each advertised
115   prefix using technologies such as L3VPN [9], 6PE [10], and
116   Softwire [8] using BGP label unicast (BGP-LU) technique[3]. A BGP
117   speaker then applies the path selection steps to choose the best
118   path. In modern networks, it is not uncommon to have a prefix
119   reachable via multiple edge routers. In addition to proprietary
120   techniques, multiple techniques have been proposed to allow for
121   BGP to advertise more than one path for a given prefix
122   [7][12][13], whether in the form of equal cost multipath or
123   primary-backup. Another common and widely deployed scenario is
124   L3VPN with multi-homed VPN sites with unique Route Distinguisher.
125   It is advantageous to utilize the commonality among paths used by
126   NLRIs[1] to significantly improve convergence in case of topology
127   modifications.

[] I still haven't read the rest of the document...  PIC is a valuable and
simple technique.  Just from the first sentence (and peeking ahead at some
of the figures and examples), it seems to me that the description could
have been significantly simpler. :-(


[major] rfc7322 requires that RFCs be referenced as [RFCXXXX], please
update the citations.


[minor nit] Multiple citations don't have spaces around the brackets
("[]").


[minor] The reference to rfc4271 is the general reference for BGP, no need
to include a reference to rfc4760 with it.


[] "for labeled address families...L3VPN with multi-homed VPN sites with
unique Route Distinguisher"

The detail about labeled AFs/L3VPN seems unnecessary given that PIC doesn't
just apply to them.


[major] "BGP label unicast (BGP-LU) technique[3]"

Note that rfc8277 doesn't use the terms BGP-LU, "label unicast", or
"labeled unicast".  I know that BGP-LU is commonly used, but it is not
defined in any IETF document (that I could find).  Please define it in the
terminology section.  Also, don't refer to rfc8277 every time you use the
term.


[major] "A BGP speaker then applies the path selection steps to choose the
best path."

rfc4271 doesn't talk about "path selection" or finding a "best path".  It
specifies a Decision Process that results in best routes.  Please use
consistent terminology.


[minor] "In addition to proprietary techniques..."

These proprietary techniques are not described anywhere.  Please don't
mention them -- it will only raise unnecessary questions.


[nit] "techniques have been proposed...[7][12][13]"

Add-path and Diverse paths are not just "proposed".


[minor] "It is advantageous to utilize the commonality among paths used by
NLRIs[1] to significantly improve convergence in case of topology
modifications."

You may have to be more explicit about which commonality -- mention it
explicitly.



129   This document proposes a hierarchical and shared forwarding chain
130   organization that allows traffic to be restored to pre-calculated
131   alternative equal cost primary path or backup path in a time
132   period that does not depend on the number of BGP prefixes. The
133   technique relies on internal router behavior that is completely
134   transparent to the operator and can be incrementally deployed and
135   enabled with zero operator intervention. In other words, once it
136   is implemented and deployed on a router, nothing is required from
137   the operator to make it work. It is noteworthy to mention that
138   this document describes FIB architecture that can be implemented
139   in both hardware and/or software.

[nit] s/to pre-calculated/to a pre-calculated


[nit] s/describes FIB/describes a FIB


[minor] Expand FIB on first mention.



141 1.1. Terminology

[major] Please don't deviate (at least) from terminology used in standards
track specifications.  Please don't redefine terms to mean something
different in this document.

If using terminology defined elsewhere, please just reference those
documents, and don't redefine the terms here.



143   This section defines the terms used in this document. For ease of
144   use, we will use terms similar to those used by L3VPN [9].

[major] Similar?  If you're using the terminology from rfc4364, then please
just use it.  Redefining terms doesn't make anything easier.

To be clear: remove any terms that are already defined in rfc4364 and make
the reference Normative.



146   o  BGP prefix: A prefix P/m (of any AFI/SAFI) that a BGP speaker
147      has a path for.

[major] rfc4271 uses the concept of an "IP prefix" carried in a "route" and
described by a "path" (from §1.1):

   Route
      A unit of information that pairs a set of destinations with the
      attributes of a path to those destinations.  The set of
      destinations are systems whose IP addresses are contained in one
      IP address prefix carried in the Network Layer Reachability
      Information (NLRI) field of an UPDATE message.  The path is the
      information reported in the path attributes field of the same
      UPDATE message.



149   o  IGP prefix: A prefix P/m (of any AFI/SAFI) that is learnt via
150      an Interior Gateway Protocol (IGP), such as OSPF and ISIS. The
151      prefix may be learnt directly through the IGP or redistributed
152      from other protocol(s).

[major] rfc4271 uses the term "IGP route".


[minor] The "P/m" notation is not explained -- and is not used outside of
this section.


[major] "is learnt via an...IGP...may be learnt directly through the IGP or
redistributed from other protocol(s)"

There's a conflict in the definition: does it come directly from the IGP or
from something else?  Might might those "other protocol(s)" be?



154   o  CE[7]: An external router through which an egress PE can reach
155      a prefix P/m.

[minor] Please expand CE.


[major] The reference to draft-ietf-idr-best-external seems incorrect; that
draft doesn't use CE at all.  This seems to be a term that comes from
RFC4364.


[minor] Please expand PE in first use.



157   o  Egress PE[7], "ePE": A BGP speaker that learns about a prefix
158      through an eBGP peer and chooses that eBGP peer as the next-hop
159      for that prefix.

[major] The reference to draft-ietf-idr-best-external seems incorrect; that
draft doesn't use PE at all.  This seems to be a term that comes from
RFC4364.


[minor] s/eBGP/EBGP/g
That is how rfc4271 uses it.


[minor] Expand EBGP on first use.



161   o  Ingress PE, "iPE": A BGP speaker that learns about a prefix
162      through a iBGP peer and chooses an egress PE as the next-hop for
163      the prefix.

[minor] s/iBGP/IBGP/g
That is how rfc4271 uses it.


[minor] Expand IBGP on first use.



165   o  Path: The next-hop in a sequence of nodes starting from the
166      current node and ending with the destination node or network
167      identified by the prefix. The nodes may not be directly
168      connected.

[major] See the definition above from rfc4271.


[major] Some of the definitions refer to BGP, but others to the FIB
characteristics....differentiate.  "FIB Path" vs "BGP Path", for example.
Note that rfc4271 uses the term "Routing Table" as follows (from §3.2:
Routing Information Base):

   Routing information that the BGP speaker uses to forward packets (or
   to construct the forwarding table used for packet forwarding) is
   maintained in the Routing Table.  The Routing Table accumulates
   routes to directly connected networks, static routes, routes learned
   from the IGP protocols, and routes learned from BGP.  Whether a
   specific BGP route should be installed in the Routing Table, and
   whether a BGP route should override a route to the same destination
   installed by another source, is a local policy decision, and is not
   specified in this document.  In addition to actual packet forwarding,
   the Routing Table is used for resolution of the next-hop addresses
   specified in BGP updates (see Section 5.1.3).



170   o  Recursive path: A path consisting only of the IP address of the
171      next-hop without the outgoing interface. Subsequent lookups are
172      necessary to determine the outgoing interface and a directly
173      connected next-hop.

[major] rfc4271 already used "recursive route".



175   o  Non-recursive path: A path consisting of the IP address of a
176      directly connected next-hop and outgoing interface.

[] See above.



...
181   o  Primary path: A recursive or non-recursive path that can be
182      used all the time as long as a walk starting from this path can
183      end to an adjacency. A prefix can have more than one primary
184      path.

[minor] "can be used all the time"

I'm not sure what you mean here.


[minor] The concept of a "walk" hasn't been introduced -- please put a
pointer to where that is discussed.



...
189   o  Leaf: A container data structure for a prefix or local label.
190      Alternatively, it is the data structure that contains prefix
191      specific information.

[] So a leaf can be many things: "prefix or local label...[or] prefix
specific information".  Hmmm....



...
198   o  Pathlist: An array of paths used by one or more prefixes to
199      forward traffic to destination(s) covered by an IP prefix. Each
200      path in the pathlist carries its "path-index" that identifies its
201      position in the array of paths. In general, the value of the
202      "path-index" stored in path may not necessarily have the same
203      value of the location of the path in the pathlist. For example
204      the 3rd path may carry path-index value of 1. A pathlist may
205      contain a mix of primary and backup paths.

[nit] s/its "path-index"/a "path-index"



207   o  OutLabel-List: Each labeled prefix is associated with an
208      OutLabel-List. The OutLabel-List is an array of one or more
209      outgoing labels and/or label actions where each label or label
210      action has 1-to-1 correspondence to a path in the pathlist.
211      Label actions[6] are: push the label, pop the label, swap the
212      incoming label with the label in the OutLabel-List entry, or
213      don't push anything at all in case of "unlabeled". The prefix
214      may be an IGP or BGP prefix.

[major] If the label label actions are defined in rfc3031, please just
point to it -- is the "swap the incoming label with the label in the
OutLabel-List entry" defined there too?

It seems to me that the action defined here is part of a general action
taken by the FIB, and not specific only to labels (as in a different
encapsulation, for example).



216   o  Forwarding chain: It is a compound data structure consisting of
217      multiple connected block that a forwarding engine walks one
218      block at a time to forward the packet out of an interface.
219      Section 2.2 explains an example of a forwarding chain.
220      Subsequent sections provide additional examples

[nit] s/multiple connected block/multiple connected blocks


[nit] s/additional examples/additional examples.


[minor] Note that both "forwarding chain" and "forwarding Chain" are used
throughout the text.  Please be consistent.



...
229   o  Route: A prefix with one or more paths associated with it.  The
230      minimum set of objects needed to construct a route is a leaf
231      and a pathlist.

[major] See the comments above about rfc4271 and the FIB in general.



233 2. Overview

235   The idea of BGP-PIC is based on two pillars

[minor] The descriptions below are very thick (hard to read through)
without more context.  Suggestion: keep the description simple in this
section and point to where there is more detail.



236   o  A shared hierarchical forwarding chain: It is not uncommon to see
237      multiple destinations reachable via the same list of next-hops.
238      Instead of having a separate list of next-hops for each
239      destination, all destinations sharing the same list of next-hops
240      can point to a single copy of this list thereby allowing fast
241      convergence by making changes to a single shared list of next-
242      hops rather than possibly a large number of destinations. Because
243      paths in a pathlist may be recursive, a hierarchy is formed
244      between pathlist and the resolving prefix whereby the pathlist
245      depends on the resolving prefix.

247   o  A forwarding plane that supports multiple levels of indirection:
248      A forwarding chain that starts with a destination and ends with
249      an outgoing interface is not a simple flat structure. Instead a
250      forwarding entry is constructed via multiple levels of
251      dependency. A BGP NLRI uses a recursive next-hop, which in turn
252      resolves via an IGP next-hop, which in turn resolves via an
253      adjacency consisting of one or more outgoing interface(s) and
254      next-hop(s).

[nit] s/Instead a/Instead, a


[minor] Are "multiple levels of indirection" and "multiple levels of
dependency" the same thing?  The latter is used to describe the former.


[] Next-hop is also a term that needs to be differentiated.  In this
paragraph is it used in the context of a BGP route, an IGP route, and a
local address on the FIB.  See rfc4271.



256   Designing a forwarding plane that constructs multi-level forwarding
257   chains with maximal sharing of forwarding objects allows rerouting a
258   large number of destinations by modifying a small number of objects
259   thereby achieving convergence in a time frame that does not depend
260   on the number of destinations. For example, if the IGP prefix that
261   resolves a recursive next-hop is updated there is no need to update
262   the possibly large number of BGP NLRIs that use this recursive next-
263   hop.

[] The example is orders of magnitue simpler that the actual explanation.
BTW, none of the words used ("multi-level forwarding chains with maximal
sharing of forwarding objects") are mentioned in the description of the
pillars.



265 2.1. Dependency

267   This section describes the required functionalities in the
268   forwarding and control planes to support BGP-PIC described in this
269   document.

[nit] s/BGP-PIC described/BGP-PIC as described



271 2.1.1. Hierarchical Hardware FIB (Forwarding Information Base)

[major] The Introduction says that "this document describes FIB
architecture that can be implemented in both hardware and/or software", but
this section requires a hardware implementation.  ??



273   BGP-PIC requires a hierarchical hardware FIB support: for each BGP
274   forwarded packet, a BGP leaf is looked up, then a BGP Pathlist is
275   consulted, then an IGP Pathlist, then an Adjacency.

[major] "BGP forwarded packet"

I think you mean something along the lines of a packet destined to an
external destination, or to a destination learned through BGP, or ...  ??


[major] The definitions of both a leaf and a pathlist don't assume protocol
origin.  IOW, the reader has to make significant assumptions to understand
that you mean "a leaf/pathlist that resulted from BGP/IGP-learned
information" (or something like that).  If the identification of the origin
protocol is required, please also say so.


[major] "Pathlist" and "Adjancency" are capitalized, but "leaf" isn't.
Personally, I like capitalizing terms with specific meaning.  Not all
instances (throughout the document) are treated the same.  Please be
consistent throughout.


[major] The general process above: lookup the destination (BGP leaf), which
points to possible alternatives (BGP pathlist), which recursively
resolve...  assumes knowledge of how a FIB is generally built and the
operations that forwarding a packet requires.  This may not be common
knowledge to every reader.  Please either explain the assumptions or
reference a place where the explanation exists already.



277   An alternative method consists in "flattening" the dependencies when
278   programming the BGP destinations into HW FIB resulting in
279   potentially eliminating both the BGP Path-List and IGP Path-List
280   consultation. Such an approach decreases the number of memory
281   lookups per forwarding operation at the expense of HW FIB memory
282   increase (flattening means less sharing thereby less duplication),
283   loss of ECMP properties (flattening means less pathlist entropy) and
284   loss of BGP-PIC properties.

[minor] HW is mentioned here again...and the concept of programming, which
goes back to the general knowledge assumptions that you're making.


[minor] s/Path-List/Pathlist/g


[major] Am I to assume that this "alternative method" shouldn't be
considered/used, especially if it results in loss of the "BGP-PIC
properties"??  This section started by mentioning a requirement -- if this
other approach is not what is required the why even mention it?



286 2.1.2. Availability of more than one BGP next-hops

288   When the primary BGP next-hop fails, BGP-PIC depends on the
289   availability of one or more pre-computed and pre-installed secondary
290   BGP next-hop(s) in the BGP Pathlist.

[major] "primary BGP next-hop"

At this point in the document you've hinted at the possibility of having
multiple BGP routes (with potentially different NEXT_HOPs), but the concept
of "primary BGP next-hop" hasn't been explained.  I'm assuming that by this
you mean the BGP best route [rfc4271].  (Again, you're making assumptions
of the knowledge of the reader.)

I can see that this section talks about "secondary next-hops" later.
Perhaps change the order of the text so that the conditions precede the
explanation of the actions.


[major] "BGP next-hop fails"

rfc4271 talks about the the address in the NEXT_HOP attribute becoming not
resolvable.  Please use established terminology!


[major] "pre-computed and pre-installed...in the BGP Pathlist"

There's an example of "pre-computed" later on, but I couldn't find an
explanation of "pre-installed...in the BGP Pathlist".



292   The existence of a secondary next-hop is clearly required for the
293   following reason: a service caring for network availability will
294   require two disjoint network connections resulting in two BGP next-
295   hops.

[minor] You mean "clearly required" for BGP-PIC, right?  Not for the
service.



297   The BGP distribution of secondary next-hops is available thanks to
298   the following BGP mechanisms: Add-Path [12], BGP Best-External [7],
299   diverse path [13], and the frequent use in VPN deployments of
300   different VPN RD's per PE. Another option to learn multiple BGP
301   NH/path is simply to receive IBGP paths from multiple BGP RR
302   selection a different path as best. It is noteworthy to mention that
303   the availability of another BGP path does not mean that all failure
304   scenarios can be covered by simply forwarding traffic to the
305   available secondary path. The discussion of how to cover various
306   failure scenarios is beyond the scope of this document.

[minor] "BGP NH/path" is a new term.  Even if using "NH" seems obvious,
please mention it at least once.  Also, you've been talking (incorrectly)
about BGP paths (instead of routes) or BGP next-hops and now you're
combining the two.  This is the only instance.


[nit] s/is simply to receive/is to receive


[nit] s/multiple BGP RR selection/multiple BGP RRs selecting


[minor] You might want to reference rfc9107 as an example of "multiple BGP
RRs selecting a different path as best".



308 2.2. BGP-PIC Illustration

310   To illustrate the two pillars above as well as the platform
311   dependency, we will use an example of a simple multihomed L3VPN [9]
312   prefix in a BGP-free core running LDP [4] or segment routing over
313   MPLS forwarding plane [5].

[nit] s/a simple multihomed/a multihomed

What is "simple" to you may not be simple to others.  Just state the facts.


[minor] s/L3VPN [9]/L3VPN
The citation is not needed all the time.


[minor] "core running LDP [4] or segment routing over MPLS forwarding plane
[5]"

This is an example.  To simplify, pick one.



315    +--------------------------------+
316    |                                |
317    |                               ePE2 (IGP-IP1 192.0.2.1, Loopback)
318    |                                |  \
319    |                                |   \
320    |                                |    \
321   iPE                               |    CE....VRF "Blue", ASnum 65000
322    |                                |    /   (VPN-IP1 198.51.100.0/24)
323    |                                |   /    (VPN-IP2 203.0.113.0/24)
324    |   LDP/Segment-Routing Core     |  /
325    |                               ePE1 (IGP-IP2 192.0.2.2, Loopback)
326    |                                |
327    +--------------------------------+
328             Figure 1 VPN prefix reachable via multiple PEs

[nit] s/ASnum/ASN

Expand on first use.



330   Referring to Figure 1, suppose the iPE (the ingress PE) receives
331   NLRIs for the VPN prefixes VPN-IP1 and VPN-IP2 from two egress PEs,
332   ePE1 and ePE2 with next-hop BGP-NH1 and BGP-NH2, respectively.
333   Assume that ePE1 advertise the VPN labels VPN-L11 and VPN-L12 while
334   ePE2 advertise the VPN labels VPN-L21 and VPN-L22 for VPN-IP1 and
335   VPN-IP2, respectively. Suppose that BGP-NH1 and BGP-NH2 are resolved
336   via the IGP prefixes IGP-IP1 and IGP-IP2, where each happen to have
337   2 equal cost paths with IGP-NH1 and IGP-NH2 reachable via the
338   interfaces I1 and I2, respectively. Suppose that local labels
339   (whether LDP [4] or segment routing [5]) on the downstream LSRs[4]
340   for IGP-IP1 are IGP-L11 and IGP-L12 while for IGP-IP2 are IGP-L21
341   and IGP-L22. As such, the routing table at iPE is as follows:

[minor] "receives NLRIs"

I other places you're talked about paths or routes...please be consistent!


[minor] "IGP-NH1 and IGP-NH2 reachable via the interfaces I1 and I2"

I'm sure you're talking about interfaces on the iPE, but others may find
that the picture is lacking detail when compared to the description: the
core is just a box..



343         65000: 198.51.100.0/24
344            via ePE1 (192.0.2.1), VPN Label: VPN-L11
345            via ePE2 (192.0.2.2), VPN Label: VPN-L21

[minor] You need to explain the notation and, for example, why some entries
have an ASN and others don't.


[minor] It might help if the description included the IP addresses.  For
example, when mentioning BGP-NH1, indicate that it refers to 192.0.2.1.



...
362   IP Leaf:    Pathlist:    IP Leaf:           Pathlist:
363   --------  +-------+     --------          +----------+
364   VPN-IP1-->|BGP-NH1|-->IGP-IP1(BGP NH1)--->|IGP-NH1,I1|--->Adjacency1
365     |       |BGP-NH2|-->....      |         |IGP-NH2,I2|--->Adjacency2
366     |       +-------+             |         +----------+
367     |                             |
368     |                             |
369     v                             v
370   OutLabel-List:                OutLabel-List:
371   +----------------------+      +----------------------+
372   |VPN-L11 (VPN-IP1, NH1)|      |IGP-L11 (IGP-IP1, NH1)|
373   |VPN-L21 (VPN-IP1, NH2)|      |IGP-L12 (IGP-IP1, NH2)|
374   +----------------------+      +----------------------+

376           Figure 2 Shared Hierarchical Forwarding Chain at iPE

378   The forwarding chain depicted in Figure 2 illustrates the first
379   pillar, which is sharing and hierarchy. We can see that the BGP
380   pathlist consisting of BGP-NH1 and BGP-NH2 is shared by all NLRIs
381   reachable via ePE1 and ePE2. As such, it is possible to make changes
382   to the pathlist without having to make changes to the NLRIs. For
383   example, if BGP-NH2 becomes unreachable, there is no need to modify
384   any of the possibly large number of NLRIs. Instead only the shared
385   pathlist needs to be modified. Likewise, due to the hierarchical
386   structure of the forwarding chain, it is possible to make
387   modifications to the IGP routes without having to make any changes
388   to the BGP NLRIs. For example, if the interface "I2" goes down, only
389   the shared IGP pathlist needs to be updated, but none of the IGP
390   prefixes sharing the IGP pathlist nor the BGP NLRIs using the IGP
391   prefixes for resolution need to be modified.

[minor] "We can see that the BGP pathlist consisting of BGP-NH1 and BGP-NH2
is shared by all NLRIs reachable via ePE1 and ePE2."

I only see VPN-IP1 using that pathlist.

I know it is not easy to draw using ASCII, but you may want to provide a
conceptual figure showing how the leafs and Pathlists relate to each other
and then describe the contents.  OR  You can also integrate an SVI drawing
in the XML.


[minor] The representation of the Pathlists is not consistent: the first
one shows just BGP-NHs, while the second one has an extra element.  It may
be obvious to you that the interface element points at the corresponding
Adjacencies, but please explain that too.


[major] The description is not complete -- it jumps from using one Pathlist
to the other, but it doesn't cover the recursive nature of the resolution:
resolving the BGP-NHs in the Pathlist through another leaf.


[major] The outlabel-list is not explained anywhere.



393   Figure 2 can also be used to illustrate the second BGP-PIC pillar.
394   Having a deep forwarding chain such as the one illustrated in Figure
395   2 requires a forwarding plane that is capable of accessing multiple
396   levels of indirection in order to calculate the outgoing
397   interface(s) and next-hops(s). While a deeper forwarding chain
398   minimizes the re-convergence time on topology change, there will
399   always exist platforms with limited capabilities and hence imposing
400   a limit on the depth of the forwarding chain. Section 5 describes
401   how to gracefully trade off convergence speed with the number of
402   hierarchical levels to support platforms with different
403   capabilities.

[minor] "Figure 2 can also be used to illustrate the second BGP-PIC
pillar."

Ok.  But the rest of the paragraph doesn't explain how that is needed in
the example.



405   Another example using IPv6 addresses can be something like the
406   following

408         65000: 2001:DB8:1::/48
409            via ePE1 (192::1), VPN Label: VPN6-L11
410            via ePE2 (192::2), VPN Label: VPN6-L21

412         65000: 2001:DB8:2:/48
413            via ePE1 (192::1), VPN Label: VPN6-L12
414            via ePE2 (192::2), VPN Label: VPN6-L22

416         192::1/128
417            via Core, Label:  IGP6-L11
418            via Core, Label:  IGP6-L12

420         192::2/128
421            via Core, Label:  IGP6-L21
422            via Core, Label:  IGP6-L22

424   The same hierarchical forwarding chain described can be constructed
425   for IPv6 addresses/prefixes.

[major] Please use the addresses from rfc3849.


[] BTW, sorry to be blunt, but this is a lazy attempt at using IPv6 in an
example.  You should at least show a figure and maybe a different
scenario...not just similar numbers.



427 3. Constructing the Shared Hierarchical Forwarding Chain

429   Constructing the forwarding chain is an application of the two
430   pillars described in Section 2. This section describes how to
431   construct the forwarding chain in hierarchical shared manner.

[nit] s/in hierarchical/in a hierarchical



433 3.1. Constructing the BGP-PIC forwarding Chain

435   The whole process starts when BGP downloads a prefix to FIB. The
436   prefix contains one or more outgoing paths. For certain labeled
437   prefixes, such as VPN [9] prefixes, each path may be associated with
438   an outgoing label and the prefix itself may be assigned a local
439   label. The list of outgoing paths defines a pathlist. If such
440   pathlist does not already exist, then FIB manager (software or
441   hardware entity responsible for managing the FIB) creates a new
442   pathlist, otherwise the existing pathlist is used. The BGP prefix is
443   added as a dependent of the pathlist.

[major] "BGP downloads a prefix to FIB"

rfc4271 talks about "routes...installed in the Routing Table".  Please, as
much as possible, use the terminology from established RFCS.  Note that
this document doesn't define what a FIB is.


[major] "The prefix contains one or more outgoing paths."

This may be another terminology issue: a BGP route doesn't contain multiple
next-hops.  If the routes came from ADD_PATH or ORR, etc., there will be
multiple BGP routes with a common NLRI and different next-hops.  IOW, it
looks like the Pathlist would be built incrementally as routes for the same
NLRI are installed.

The end result should be the same.  Terminology matters!


[minor] "VPN [9]"

The other references to rfc4364 use "L3VPN", should this one use it too?


[major] "If such pathlist does not already exist..."

It should be mentioned somewhere the reason for this Pathlist to already
exist: there's probably another BGP route using it.  This is where the
connection to the shared nature of the chain needs to be made.  Otherwise,
the description sounds as if there might simply be a Pathlist structure
available...


[nit] s/FIB manager/the FIB manager/g


[] It might be easier/better/clearer to explain the process as a series of
steps.  Note that the description above is really a series of steps: BGP
has a route to install, the route is accepted in the routing table (local
policy), a leaf for this NLRI exists (or not), the Pathlist exists (or
not), etc...



445   The previous step constructs the upper part of the hierarchical
446   forwarding chain. The forwarding chain is completed by resolving the
447   paths of the pathlist. A BGP path usually consists of a next-hop.
448   The next-hop is resolved by finding a matching IGP prefix.

[major] "The forwarding chain is completed by resolving the paths of the
pathlist."

The example in the last section shows that this "lower part" (is that the
right term?) could also include a Pathlist.  The explanation of how the
"upper part" is constructed didn't mention that the process was to
"resolve" the BGP NLRI.  As a result, there's a disconnect between what was
done before and what is assumed here.

Again, a series of steps would serve the purpose better.

Also, be mindful of the use of "resolve" in rfc4271.


[major] "A BGP path usually consists of a next-hop."

As opposed to what?  I mean the NEXT_HOP attribute is mandatory, a route
includes multiple other Path Attributes...  Not sure what the point it.


[minor] "The next-hop is resolved by finding a matching IGP prefix."

Yes, as an extreme simplification.  §9.1.2.1/rfc4271 offers a better
description.



450   The end result is a hierarchical shared forwarding chain where the
451   BGP pathlist is shared by all BGP prefixes that use the same list of
452   paths and the IGP prefix is shared by all pathlists that have a path
453   resolving via that IGP prefix. It is noteworthy to mention that the
454   forwarding chain is constructed without any operator intervention at
455   all.

[minor] "The end result is a hierarchical shared forwarding chain where the
BGP pathlist is shared..."

As I mentioned above, more emphasis should be places on (when explaining
the steps) the shared nature of the Pathlists and how its dependents are
independent of each other too.


[minor] "without any operator intervention"

Yes, this was mentioned as an advantage somewhere.  It would be better if
the document was explicit from the start on the point that it is describing
a set of abstract structures and procedures to be internally instantiated.
Just like protocol structures (neighbors, routes, etc.) that are
transparent to the user.



...
460 3.2. Example: Primary-Backup Path Scenario

462   Consider the egress PE ePE1 in the case of the multi-homed VPN
463   prefixes in the BGP-free core depicted in Figure 1. Suppose ePE1
464   determines that the primary path is the external path but the backup
465   path is the iBGP path to the other PE ePE2 with next-hop BGP-NH2.
466   ePE2 constructs the forwarding chain depicted in Figure 3. We are
467   only showing a single VPN prefix for simplicity. But all prefixes
468   that are multihomed to ePE1 and ePE2 share the BGP pathlist.

[minor] "BGP-free core depicted in Figure 1"

It doesn't really make a difference, but it wasn't mentioned before that
Figure 1 has a BGP-free core.  I'm mentioning this because there are
inconsistencies in the description.  Also, this is a detail that is not
important to the description, but that can create confusion for other
readers and generate unnecessary questions and comments.


[minor] "Suppose ePE1 determines..."

It would be great if there was either a pointer to how ePE1 does all this
(where are backups specified?), or a little bit more explanation of the
assumptions.


[nit] s/external path but the backup/external path, while the backup



470                    BGP OutLabel-List
471     VPN-L11            +---------+
472   (Label-leaf)---+---->|Unlabeled|
473                  |     +---------+
474                  v     | VPN-L21 |
475                  |     | (swap)  |
476                  |     +---------+
477                  |
478                  |                    BGP Pathlist
479                  |                   +------------+    Connected route
480                  |                   |   CE-NH    |------>(to the CE)
481                  |                   |path-index=0|
482                  |                   +------------+
483                  |                   |  VPN-NH2   |
484     VPN-IP1 -----+------------------>|  (backup)  |------>IGP Leaf
485   (IP leaf)                          |path-index=1|    (Towards ePE2)
486        |                             +------------+
487        |
488        |           BGP OutLabel-List
489        |              +---------+
490        +------------->|Unlabeled|
491                       +---------+
492                       | VPN-L21 |
493                       | (push)  |
494                       +---------+

496   Figure 3 : VPN Prefix Forwarding Chain with eiBGP paths on egress PE

[minor] Figure 2 provided an easy-to-follow dependency order: Leaf to
Pathlist  to Leaf to Pathlist.  The dependencies in Figure 3 are not
straightforward, please provide a description of the figure before
explaining the differences with respect to the other example.


[major] The definition of Pathlist (§1.1) implies that all its entries can
be used for forwarding, but the Pathlist above (and this use case) clearly
indicate that not all the entries in a Pathlist are used in the same way,
or at the same time.



498   The example depicted in Figure 3 differs from the example in Figure
499   2 in two main aspects. First, as long as the primary path towards
500   the CE (external path) is useable, it will be the only path used for
501   forwarding while the OutLabel-List contains both the unlabeled label
502   (primary path) and the VPN label (backup path) advertised by the
503   backup path ePE2. The second aspect is presence of the label leaf
504   corresponding to the VPN prefix. This label leaf is used to match
505   VPN traffic arriving from the core. Note that the label leaf shares
506   the pathlist with the IP prefix.

[major] "the...path...is useable"

What does "useable" mean?  Please describe this term in the terminology
section.


[major] "primary path towards the CE (external path) is useable, it will be
the only path used"

The arrow from VPN-IP1 points to the VPN-NH2 entry in the Pathlist, so the
description doesn't seem to match.


[major] "the OutLabel-List"

There are two boxes named "BGP OutLabel-List", and they contain different
backup actions (swap vs push).


[minor] "unlabeled label"

rfc3031 talks about unlabeled packets, not labels.


[minor] A third difference is that this figure includes a "path-index" --
but that is not explained.



508 4. Forwarding Behavior
...
513   When a packet arrives at a router, it matches a leaf. A labeled
514   packet matches a label leaf while an IP packet matches an IP leaf.
515   The forwarding engines walks the forwarding chain starting from the
516   leaf until the walk terminates on an adjacency. Thus when a packet
517   arrives, the chain is walked as follows:

[minor] Instead of summarizing, simply start with the steps.



519   1. Lookup the leaf based on the destination address or the label at
520      the top of the packet.

[major] Lookup how?  How do you know you found the right leaf?

I think rfc3031 has some text related to the forwarding of labeled
packets.  But I don't know of a reference for IP forwarding, so you should
at least mention something around a longest-prefix-match.



522   2. Retrieve the parent pathlist of the leaf.

[minor] While it is related, the terminology section defines the dependency
relationship as the leaf being the child of the Pathlist -- keeping
consistency is important.



524   3. Pick the outgoing path "Pi" from the list of resolved paths in
525      the pathlist. The method by which the outgoing path is picked is
526      beyond the scope of this document (e.g. flow-preserving hash
527      exploiting entropy within the MPLS stack and IP header). Let the
528      "path-index" of the outgoing path "Pi" be "j".

[minor] s/Pick the outgoing path "Pi"/Pick an outgoing path "Pi"


[] "Let the "path-index" of the outgoing path "Pi" be "j"."

I'm not sure if you mean that "j is the path-index of Pi" or "assign a
path-index of j".  If the former, how is the path-index assigned?



530   4. If the prefix is labeled, use the "path-index" "j" to retrieve
531      the jth label "Lj" stored the jth entry in the OutLabel-List and
532      apply the label action of the label on the packet (e.g. for VPN
533      label on the ingress PE, the label action is "push"). As
534      mentioned in Section 1.1 the value of the "path-index" stored
535      in path may not necessarily be the same value of the location of
536      the path in the pathlist.

[minor] "retrieve the jth label "Lj" stored the jth entry in the
OutLabel-List"

This sounds as if there were (at least) j labels in each of (at least) j
entries in the OutLabel-List.   But I assume that there is really only one
label (or label stack) in that entry.   ??



538   5. Move to the parent of the chosen path "Pi".

540   6. If the chosen path "Pi" is recursive, move to its parent prefix
541      and go to step 2.

543   7. If the chosen path is non-recursive move to its parent adjacency.
544      Otherwise go to the next step.

[minor] These 2 steps say the same thing: "move to the parent of..."Pi"",
"path "Pi"...move to its parent", and "move to its parent".  I think you
need to better differentiate (?) the options.



...
549   Let's apply the above forwarding steps to the forwarding chain
550   depicted in Figure 2 in Section 2. Suppose a packet arrives at
551   ingress PE iPE from an external neighbor. Assume the packet matches
552   the VPN prefix VPN-IP1. While walking the forwarding chain, the
553   forwarding engine applies a hashing algorithm to choose the path and
554   the hashing at the BGP level yields path 0 while the hashing at the
555   IGP level yields path 1. In that case, the packet will be sent out
556   of interface I2 with the label stack "IGP-L12,VPN-L11".

[minor] s/path 0...path 1/path-index 0...path-index 1


[minor] The example would be better understood if it was presented as steps
matching the process described above (vs a narrative explanation).  Also,
be specific in which path is the one with each path-index...


[major] Figure 2 doesn't contain any indication of a path-index.

558 5. Handling Platforms with Limited Levels of Hierarchy


[] This whole section, which is longer than the text describing the
"normal" mechanism (!), seems to be more appropriate for an Appendix than
the main body of the document.  Please move it.



...
564   o  Being able to reduce the number of hierarchical levels from any
565      arbitrary value to a smaller arbitrary value that can be
566      supported by the forwarding engine.

[?] What is the worst case in terms of number of levels?  3?  Just curious.
I'm asking simply because an "arbitrary value" sounds like it could be 10s
o even 1000s, when in reality it is not more than a handful.  Are there
many platforms out there that wouldn't support 2-3 levels (for the number
of routes/paths they normally would carry)?


[minor] "smaller arbitrary value"

This smaller value is not really arbitrary (as in random), it is determined
by the platform.  Maybe something along the lines of a "value supported by
the platform" would be better.



...
571 5.1. Flattening the Forwarding Chain
...
584   1. The forwarding engine is now at leaf "R1".
...
605   9. Now the forwarding engine continues the walk to the parent of
606      "Qj".

[minor] This list of steps is just a restatement of the process in the last
section.  IMO there's no need to repeat the steps here.  Point at the
2-level example, and jump directly to the next paragraph.

To reference from the explanation below, "a picture is worth a thousand
words". ;-)



...
614   1. FIB manager wants to reduce the number of levels used by "Pi" by
615      1.

[nit] This is not really a step...



...
622   4. FIB manager also extracts the OutLabel-list(R2) associated with
623      the leaf "R2". Remember that OutLabel-list(R2) = <L1, L2,...,
624      Lm>.

[] New syntax: "OutLabel-list(R2)".



...
643   It is noteworthy to mention that the labels in the OutLabel-list
644   associated with the "flattened" pathlist may be stored in the same
645   memory location as the path itself to avoid additional memory
646   access. But that is an implementation detail that is beyond the
647   scope of this document.

[] Why even mention it then?



649   The same steps can be applied to all paths in the pathlist <P1,
650   P2,..., Pn> so that all paths are "flattened" thereby reducing the
651   number of hierarchical levels by one. Note that that "flattening" a
652   pathlist pulls in all paths of the parent paths, a desired feature
653   to utilize all ECMP/UCMP paths at all levels. A platform that has a
654   limit on the number of paths in a pathlist for any given leaf may
655   choose to reduce the number paths using methods that are beyond the
656   scope of this document.

[minor] Expand UCMP.



...
663   Because a flattened pathlist may have an associated OutLabel-list
664   the forwarding behavior has to be slightly modified. The
665   modification is done by adding the following step right after step 4
666   in Section 4.

668   5. If there is an OutLabel-list associated with the pathlist, then
669      if the path "Pi" is chosen by the hashing algorithm, retrieve the
670      label at location "i" in that OutLabel-list and apply the label
671      action of that label on the packet.

[major] Step 4 says this:

   4. If the prefix is labeled, use the "path-index" "j" to retrieve
      the jth label "Lj" stored the jth entry in the OutLabel-List and
      apply the label action of the label on the packet...

Besides using "j" instead of "i" as the path-index, both seem to say the
same thing.  Am I missing something?



...
676 5.2. Example: Flattening a forwarding chain.

678   This example uses a case of inter-AS option C [9] where there are 3
679   levels of hierarchy. Figure 4 illustrates the sample topology. To
680   force 3 levels of hierarchy, the ASBRs[9] on the ingress domain
681   (domain 1) advertise the core routers of the egress domain (domain
682   2) to the ingress PE (iPE) via BGP-LU [3] instead of redistributing
683   them into the IGP of domain 1. The end result is that the ingress PE
684   (iPE) has 2 levels of recursion for the VPN prefixes VPN-IP1 and
685   VPN-IP2.

[] Suggestion:

OLD>
   To force 3 levels of hierarchy, the ASBRs[9] on the ingress
   domain (domain 1) advertise the core routers of the egress
   domain (domain 2) to the ingress PE (iPE) via BGP-LU [3]
   instead of redistributing them into the IGP of domain 1.

NEW>
   The ASBRs on the ingress domain (Domain 1) use BGP to advertise
   the core routers (ASBRs and ePEs) of the egress domain (Domain
   2) to the iPE.


[minor] Expand ASBR on first use.  No need to add a reference.



687       Domain 1                 Domain 2
688   +-------------+          +-------------+
689   |             |          |             |
690   | LDP/SR Core |          | LDP/SR core |
691   |             |          |             |
692   |     (192.0.2.4)        |             |
693   |         ASBR11-------ASBR21........ePE1(192.0.2.1)
694   |             | \      / |   .      .  |\
695   |             |  \    /  |    .    .   | \
696   |             |   \  /   |     .  .    |  \
697   |             |    \/    |      ..     |   \VPN-IP1(198.51.100.0/24)
698   |             |    /\    |      . .    |   /VRF "Blue" ASn: 65000
699   |             |   /  \   |     .   .   |  /
700   |             |  /    \  |    .     .  | /
701   |             | /      \ |   .       . |/
702   iPE        ASBR12-------ASBR22........ePE2 (192.0.2.2)
703   |     (192.0.2.5)        |             |\
704   |             |          |             | \
705   |             |          |             |  \
706   |             |          |             |   \VRF "Blue" ASn: 65000
707   |             |          |             |   /VPN-IP2(203.0.113.0/24)
708   |             |          |             |  /
709   |             |          |             | /
710   |             |          |             |/
711   |         ASBR13-------ASBR23........ePE3(192.0.2.3)
712   |     (192.0.2.6)        |             |
713   |             |          |             |
714   |             |          |             |
715   +-------------+          +-------------+
716    <===========  <=========  <============
717   Advertise ePEx  Advertise   Redistribute
718   Using iBGP-LU   ePEx Using    IGP into
719                    eBGP-LU        BGP

721               Figure 4 : Sample 3-level hierarchy topology

[minor] s/ASn/ASN/g


[major] "Redistribute IGP into BGP"  ??

I'm sure you don't mean that, but probably more something like
"redistribute the ePEx routes only"...



723   We will make the following assumptions about connectivity

[nit] s/connectivity/connectivity:



725   o  In "domain 2", both ASBR21 and ASBR22 can reach both ePE1 and
726      ePE2 using the same distance.

[nit] The figure capitalizes "Domain", but the text doesn't.  Please be
consistent!


[minor] By "distance" I assume you mean "IGP metric".  Please use that term
instead.   Apply to other occurences.



...
733   We will make the following assumptions about the labels

[nit] s/labels/labels:



...
740   o  The labels advertised by ASBR11 to iPE using BGP-LU [3] for the
741      egress PEs ePE1 and ePE2 are LASBR111(ePE1) and LASBR112(ePE2),
742      respectively.

[] Why change the notation of the labels?  This new notation
("LASBR111(ePE1)") just makes the explanation more complex. :-(



...
764         65000: 198.51.100.0/24
765            via ePE1 (192.0.2.1), VPN Label: VPN-L11
766            via ePE2 (192.0.2.2), VPN Label: VPN-L21
767         65000: 203.0.113.0/24
768            via ePE2 (192.0.2.2), VPN Label: VPN-L22
769            via ePE3 (192.0.2.3), VPN Label: VPN-L32

771         192.0.2.1/32 (ePE1)
772            Via ASBR11, BGP-LU Label: LASBR111(ePE1)
773            Via ASBR12, BGP-LU Label: LASBR121(ePE1)
774         192.0.2.2/32 (ePE2)
775            Via ASBR11, BGP-LU Label: LASBR112(ePE2)
776            Via ASBR12, BGP-LU Label: LASBR122(ePE2)
777         192.0.2.3/32 (ePE3)
778            Via ASBR13, BGP-LU Label: LASBR13(ePE3)

780         192.0.2.4/32 (ASBR11)
781            via Core, Label:  IGP-L11
782         192.0.2.5/32 (ASBR12)
783            via Core, Label:  IGP-L12
784         192.0.2.6/32 (ASBR13)
785            via Core, Label:  IGP-L13

[nit] s/Via/via/g


[minor] Only some of the addresses are annotated ("192.0.2.1/32 (ePE1)").
This is helpful, but inconsistent -- both within this example and with
previous ones.



...
797   o  The index inside the pathlist entry indicates the label that will
798      be picked from the OutLabel-List associated with the child leaf
799      if that path is chosen by the forwarding engine hashing function.

[] s/index/path-index



801   OutLabel-List                                      OutLabel-List
802     For VPN-IP1                                         For VPN-IP2
803   +------------+    +--------+           +-------+   +------------+
804   |  VPN-L11   |<---| VPN-IP1|           |VPN-IP2|-->|  VPN-L22   |
805   +------------+    +---+----+           +---+---+   +------------+
806   |  VPN-L21   |        |                    |       |  VPN-L32   |
807   +------------+        |                    |       +------------+
808                         |                    |
809                         V                    V
810                    +---+---+            +---+---+
811                    | 0 | 1 |            | 0 | 1 |
812                    +-|-+-\-+            +-/-+-\-+
813                      |    \              /     \
814                      |     \            /       \
815                      |      \          /         \
816                      |       \        /           \
817                      v        \      /             \
818                 +-----+       +-----+             +-----+
819            +----+ ePE1|       |ePE2 +-----+       | ePE3+-----+
820            |    +--+--+       +-----+     |       +--+--+     |
821            v       |            /         v          |        v
822   +--------------+ |           /   +--------------+  | +-------------+
823   |LASBR111(ePE1)| |          /    |LASBR112(ePE2)|  | |LASBR13(ePE3)|
824   +--------------+ |         /     +--------------+  | +-------------+
825   |LASBR121(ePE1)| |        /      |LASBR122(ePE2)|  | OutLabel-List
826   +--------------+ |       /       +--------------+  |    For ePE3
827   OutLabel-List    |      /        OutLabel-List     |
828       For ePE1     |     /           For ePE2        |
829                    |    /                            |
830                    |   /                             |
831                    |  /                              |
832                    v /                               v
833                +---+---+  Shared Pathlist          +---+  Pathlist
834                | 0 | 1 | For ePE1 and ePE2         | 0 |  For ePE3
835                +-|-+-\-+                           +-|-+
836                  |    \                              |
837                  |     \                             |
838                  |      \                            |
839                  |       \                           |
840                  v        \                          v
841               +------+    +------+               +------+
842           +---+ASBR11|    |ASBR12+--+            |ASBR13+---+
843           |   +------+    +------+  |            +------+   |
844           v                         v                       v
845      +-------+                  +-------+              +-------+
846      |IGP-L11|                  |IGP-L12|              |IGP-L13|
847      +-------+                  +-------+              +-------+

849       Figure 5 : Forwarding Chain for hardware supporting 3 Levels

[] The OutLabel-Lists at the bottom are not tagged as such.


[] Some down arrows ("v") are missing -- on the non-vertical lines.



851   Now suppose the hardware on iPE (the ingress PE) supports 2 levels
852   of hierarchy only. In that case, the 3-levels forwarding chain in
853   Figure 5 needs to be "flattened" into 2 levels only.

855   OutLabel-List                                  OutLabel-List
856     For VPN-IP1                                    For VPN-IP2
857   +------------+    +-------+      +-------+     +------------+
858   |  VPN-L11   |<---|VPN-IP1|      | VPN-IP2|--->|  VPN-L22   |
859   +------------+    +---+---+      +---+---+     +------------+
860   |  VPN-L21   |        |              |         |  VPN-L32   |
861   +------------+        |              |         +------------+
862                         |              |
863                         |              |
864                         |              |
865          Flattened      |              |  Flattened
866          pathlist       V              V   pathlist
867                    +===+===+        +===+===+===+     +==============+
868           +--------+ 0 | 1 |        | 0 | 0 | 1 +---->|LASBR112(ePE2)|
869           |        +=|=+=\=+        +=/=+=/=+=\=+     +==============+
870           v          |    \          /   /     \      |LASBR122(ePE2)|
871    +==============+  |     \  +-----+   /       \     +==============+
872    |LASBR111(ePE1)|  |      \/         /         \    |LASBR13(ePE3) |
873    +==============+  |      /\        /           \   +==============+
874    |LASBR121(ePE1)|  |     /  \      /             \
875    +==============+  |    /    \    /               \
876                      |   /      \  /                 \
877                      |  /       +  +                  \
878                      |  +       |  |                   \
879                      |  |       |  |                    \
880                      v  v       v  v                     \
881                    +------+    +------+              +------+
882               +----|ASBR11|    |ASBR12+---+          |ASBR13+---+
883               |    +------+    +------+   |          +------+   |
884               v                           v                     v
885           +-------+                  +-------+              +-------+
886           |IGP-L11|                  |IGP-L12|              |IGP-L13|
887           +-------+                  +-------+              +-------+

[] Same comments as above.



889     Figure 6 : Flattening 3 levels to 2 levels of Hierarchy on iPE

891   Figure 6 represents one way to "flatten" a 3 levels hierarchy into
892   two levels. There are few important points:

[nit] s/are few/are a few



...
908   o  Let's take a look at the flattened pathlist used by the prefix
909      "VPN-IP2", The pathlist associated with the prefix "VPN-IP2" has
910      three entries.

[nit] s/"VPN-IP2", The/"VPN-IP2". The



912       o The first and second entry have index "0". This is because
913         both entries correspond to ePE2. Thus when hashing performed
914         by the forwarding engine results in using first or the second
915         entry in the pathlist, the forwarding engine will pick the
916         correct VPN label "VPN-L22", which is the label advertised by
917         ePE2 for the prefix "VPN-IP2".

[nit] s/using first/using the first



...
952   o  So the packet is forwarded towards the ASBR "ASBR12" and the IGP
953      label at the top will be "L12".

[minor] The figure calls the label "IGP-L12", not just "L12".  There are
other instances of this below.



...
982 6. Forwarding Chain Adjustment at a Failure

984   The hierarchical and shared structure of the forwarding chain
985   explained in the previous section allows modifying a small number of
986   forwarding chain objects to re-route traffic to a pre-calculated
987   equal-cost or backup path without the need to modify the possibly
988   very large number of BGP prefixes. In this section, we go over
989   various core and edge failure scenarios to illustrate how FIB
990   manager can utilize the forwarding chain structure to achieve BGP
991   prefix independent convergence.

[nit] s/how FIB manager/how FIB the manager



993 6.1. BGP-PIC core

[minor] The meaning of "core" and "edge" should be explained somwehre.



...
1001   When a remote link or node fails, IGP on the ingress PE receives
1002   advertisement indicating a topology change so IGP re-converges to
1003   either find a new next-hop and/or outgoing interface or remove the
1004   path completely from the IGP prefix used to resolve BGP next-hops.
1005   IGP and/or LDP download the modified IGP leaves with modified
1006   outgoing labels for labeled core.

[nit] s/IGP/the IGP/g


[nit] s/receives advertisement/receives an advertisement


[nit] s/for labeled core/for the labeled core


[] "download"

This is the second time that the "download" concept is used without
explanation.  See my comment in §3.1.



1008   When a local link fails, FIB manager detects the failure almost
1009   immediately. The FIB manager marks the impacted path(s) as unusable
1010   so that only useable paths are used to forward packets. Hence only
1011   IGP pathlists with paths using the failed local link need to be
1012   modified. All other pathlists are not impacted. Note that in this
1013   particular case there is actually no need even to backwalk to IGP
1014   leaves to adjust the OutLabel-Lists because FIB can rely on the
1015   path-index stored in the useable paths in the pathlist to pick the
1016   right label.

[nit] s/is actually no need even to/is no need to


[nit] s/to IGP leaves/to the IGP leaves


[major] The term "backwalk" is introduced here, but not explained.



1018   It is noteworthy to mention that because FIB manager modifies the
1019   forwarding chain starting from the IGP leaves only. BGP pathlists
1020   and leaves are not modified. Hence traffic restoration occurs within
1021   the time frame of IGP convergence, and, for local link failure,
1022   assuming a backup path has been precomputed, within the timeframe of
1023   local detection (e.g. 50ms). Examples of solutions that pre-
1024   computing backup paths are IP FRR [16] remote LFA [17], Ti-LFA [15]
1025   and MRT [18] or eBGP path having a backup path [11].

[major] The case on the previous paragraph deals with a local link failure
that results in the routes being "unusable", but this paragraph mentions
restoration "assuming a backup path has been precomputed" -- the
restoration timeframe also applies to the case where there is no backup
path (as explained above).

Note that the next paragraph also talks about a backup route and seems to
ignore the "unusable" case above.


[nit] s/solutions that pre- computing/solutions that can pre-compute


[nit] s/Ti-LFA/TI-LFA


[minor] Expand FRR, LFA, TI-LFA, and MRT on first use.


[nit] s/IP FRR [16] remote LFA...[18] or eBGP/IP FRR [16], remote
LFA...[18], or eBGP


[minor] "eBGP path having a backup path [11]"

I don't think that's the right reference.



1027   Let's apply the procedure mentioned in this subsection to the
1028   forwarding chain depicted in Figure 2. Suppose a remote link failure
1029   occurs and impacts the first ECMP IGP path to the remote BGP next-
1030   hop. Upon IGP convergence, the IGP pathlist used by the BGP next-hop
1031   is updated to reflect the new topology (one path instead of two). As
1032   soon as the IGP convergence is effective for the BGP next-hop entry,
1033   the new forwarding state is immediately available to all dependent
1034   BGP prefixes. The same behavior would occur if the failure was local
1035   such as an interface going down. As soon as the IGP convergence is
1036   complete for the BGP next-hop IGP route, all its BGP depending
1037   routes benefit from the new path. In fact, upon local failure, if
1038   LFA protection is enabled for the IGP route to the BGP next-hop and
1039   a backup path was pre-computed and installed in the pathlist, upon
1040   the local interface failure, the LFA backup path is immediately
1041   activated (e.g. sub-50msec) and thus protection benefits all the
1042   depending BGP traffic through the hierarchical forwarding dependency
1043   between the routes.

[minor] "Upon IGP convergence... As soon as the IGP convergence is
effective for the BGP next-hop entry,"

There's some redundancy in the text.  Also, by using different text, you're
implying that there is a difference between "IGP convergence" and "IGP
convergence...effective for the BGP next-hop entry".


[major] "The same behavior would occur if the failure was local such as an
interface going down. As soon as the IGP convergence..."

According to the text above (2 or 3 paragraphs before), IGP convergence is
not necessary in the local failure case.



...
1050 6.2.1. Adjusting forwarding Chain in egress node failure

[minor] Most of the text talks about an "egress node".  I assume that is
the same as an "edge node" -- is it?  If so, please be consistent and/or
explain somewhere that you're referring to the same thing.


1052   When an edge node fails, IGP on neighboring core nodes send route
1053   updates indicating that the edge node is no longer reachable. IGP
1054   running on the iPE instructs FIB to remove the IP and label leaves
1055   corresponding to the failed edge node from FIB. So FIB manager
1056   performs the following steps:

[major] "IGP on neighboring core nodes send route updates indicating that
the edge node is no longer reachable"

§6.1 says that "Node failures are treated as link failures."   Is that the
same in this section?

The neighbor nodes can't detect that the node is down -- they can only
detect the link being down.  The rest of the explanation assumes that there
was a single link connecting the egress node to the core, or that all the
links were reported as down.  Neither case is explained.

If there is more than one link there shouldn't be a failure in reaching the
next-hop -- except for the change in the local router(s), similar to a core
failure.



...
1073 6.2.2. Adjusting Forwarding Chain on PE-CE link Failure
...
1082   In the first case, the rest of iBGP peers will remain unaware of the
1083   link failure and will continue to forward traffic to the edge node
1084   until the edge node attached to the failed link withdraws the BGP
1085   prefixes. If the destination prefixes are multi-homed to another
1086   iBGP peer, say ePE2, then FIB manager on the edge router detecting
1087   the link failure applies the following steps (see Figure 3):

[minor] Figure 3 is an example -- it does show a sample chain.  Figure 4
may be used as a reference for the type of backup described here.



...
1100       o The label entry in OutLabel-List corresponding to the
1101          internal path to backup egress PE has swap action to the
1102          label advertised by backup egress PE.

[nit] s/has swap action/has a swap action


[nit] s/by backup/by the backup



1104       o For an arriving label packet (e.g. VPN), the top label is
1105          swapped with the label advertised by backup egress PE and the
1106          packet is sent towards that backup egress PE.

[nit] s/label packet/labeled packet


[major] Even though you're expanding on an example, it is important for the
text to be precise.

- The label mentioned is "e.g. VPN", but you're been using more specific
labels -- from figure 3, "VPN-L21"...

- The OutLabel-Lists in Figure 3 show both "VPN-L21 (swap)" and "VPN-L21
(push)".  The same label is used, only the swap action is mentioned here.
What am I missing?

- As I mentioned before, the diagram in Figure 3 is not clear.



1108   o  For unlabeled traffic, packets are simply redirected towards
1109      backup egress PE.

[nit] s/backup/the backups


[major] "ackets are simply redirected towards backup egress PE"

How?  The backup ePE is presumably not connected (which is why a label was
needed)...??



1111   In the second case where the edge router uses the IP address of the
1112   failed link as the BGP next-hop, the edge router will still perform
1113   the previous steps. But, unlike the case of next-hop self, IGP on
1114   failed edge node informs the rest of the iBGP peers that IP address
1115   of the failed link is no longer reachable. Hence the FIB manager on
1116   iBGP peers will delete the IGP leaf corresponding to the IP prefix
1117   of the failed link. The behavior of the iBGP peers will be identical
1118   to the case of edge node failure outlined in Section 6.2.1.

[nit] s/IGP on failed edge node/the IGP on the failed edge node


[nit] s/that IP address/that the IP address



1120   It is noteworthy to mention that because the edge link failure is
1121   local to the edge router, sub-50 msec convergence can be achieved as
1122   described in [11].

[] But only at the edge router -- not in other places of the network where
the IGP has to converge.

Please keep the numbers in the Appendix.  IOW, there's no need to claim
this performance here.



1124   Let's try to apply the case of next-hop self to the forwarding chain
1125   depicted in Figure 3. After failure of the link between ePE1 and CE,
1126   the forwarding engine will route traffic arriving from the core
1127   towards VPN-NH2 with path-index=1. A packet arriving from the core
1128   will contain the label VPN-L11 at top. The label VPN-L11 is swapped
1129   with the label VPN-L21 and the packet is forwarded towards ePE2.

[minor] This paragraph is out of place: you already tried to explain this
case a few paragraphs above.  Please move the text and deduplicate.



1131 6.3. Handling Failures for Flattened Forwarding Chains

1133   As explained in the in Section 5 if the number of hierarchy levels
1134   of a platform cannot support the native number of hierarchy levels
1135   of a recursive forwarding chain, the instantiated forwarding chain
1136   is constructed by flattening two or more levels. Hence a 3 levels
1137   chain in Figure 5 is flattened into the 2 levels chain in Figure 6.

[nit] s/a 3 levels/the 3-level


[nit] s/2 levels/2-level



1139   While reducing the benefits of BGP-PIC, flattening one hierarchy
1140   into a shallower hierarchy does not always result in a complete loss
1141   of the benefits of the BGP-PIC. To illustrate this fact suppose
1142   ASBR12 is no longer reachable in domain 1. If the platform supports
1143   the full hierarchy depth, the forwarding chain is the one depicted
1144   in Figure 5 and hence the FIB manager needs to backwalk one level to
1145   the pathlist shared by "ePE1" and "ePE2" and adjust it. If the
1146   platform supports 2 levels of hierarchy, then a useable forwarding
1147   chain is the one depicted in Figure 6. In that case, if ASBR12 is no
1148   longer reachable, the FIB manager has to backwalk to the two
1149   flattened pathlists and updates both of them.

1151   The main observation is that the loss of convergence speed due to
1152   the loss of hierarchy depth depends on the structure of the
1153   forwarding chain itself. To illustrate this fact, let's take two
1154   extremes. Suppose the forwarding objects in level i+1 depend on the
1155   forwarding objects in level i. If every object on level i+1 depends
1156   on a separate object in level i, then flattening level i into level
1157   i+1 will not result in loss of convergence speed. Now let's take the
1158   other extreme. Suppose "n" objects in level i+1 depend on 1 object
1159   in level i. Now suppose FIB flattens level i into level i+1. If a
1160   topology change results in modifying the single object in level i,
1161   then FIB has to backwalk and modify "n" objects in the flattened
1162   level, thereby losing all the benefit of BGP-PIC. Experience shows
1163   that flattening forwarding chains usually results in moderate loss
1164   of BGP-PIC benefits. Further analysis is needed to corroborate and
1165   quantify this statement.

[major] "The main observation is that the loss of convergence speed due to
the loss of hierarchy depth... Further analysis is needed to corroborate
and quantify this statement."

The explanation above sounds plausible.  However, if you really don't know,
and there is a dependency on the structure, and it depends on the failure,
and there's no alternative (if the platform supports less levels)...  Why
even include this information?  It feels like the equivalent of "hand
waving". :-(



1167 7. Properties

[minor] This section is really a summary.  I wonder if it would serve a
better purpose at the start of the document as a "road map".



1169 7.1. Coverage

1171   All the possible failures, except CE node failure, are covered,
1172   whether they impact a local or remote IGP path or a local or remote
1173   BGP next-hop as described in Section 6. This section provides
1174   details for each failure and how the hierarchical and shared FIB
1175   structure proposed in this document allows recovery that does not
1176   depend on number of BGP prefixes.

[major] "except CE node failure"

Isn't that covered in §6.2.2?



1178 7.1.1. A remote failure on the path to a BGP next-hop

1180   Upon IGP convergence, the IGP leaf for the BGP next-hop is updated
1181   upon IGP convergence and all the BGP depending routes leverage the
1182   new IGP forwarding state immediately. Details of this behavior can
1183   be found in Section 6.1.

[minor] Redundant text: "Upon IGP convergence, the IGP leaf for the BGP
next-hop is updated upon IGP convergence..."



1185   This BGP resiliency property only depends on IGP convergence and is
1186   independent of the number of BGP prefixes impacted.

[] "BGP resiliency property"

The rest of the text (and the name of the document!) refers to
convergence.  It isn't until this section that you change the terminology.
Please be consistent.



1188 7.1.2. A local failure on the path to a BGP next-hop

1190   Upon LFA protection, the IGP leaf for the BGP next-hop is updated to
1191   use the precomputed LFA backup path and all the BGP depending routes
1192   leverage this LFA protection. Details of this behavior can be found
1193   in Section 6.1.

[minor] Redundant text: LFA is mentioned 3 times in the first sentence.


[major] What about the case where an LFA doesn't exist?



...
1238 7.3. Automated

1240   The BGP-PIC solution does not require any operator involvement. The
1241   process is entirely automated as part of the FIB implementation.

[] Ok -- it is not as much an automation as it is an implementation choice.



1243   The salient points enabling this automation are:

[minor] I"m not sure if you're saying that the list below is what enables
the "automation", or if the "automation" results in that.



1245   o  Extension of the BGP Best Path to compute more than one primary
1246      ([12]and [13]) or backup BGP next-hop ([7] and [14]).

[major] This functionality is available regardless of the structure of the
FIB.



1248   o  Sharing of BGP Path-list across BGP destinations with same
1249      primary and backup BGP next-hop.

[nit] s/same/the same



1251   o  Hierarchical indirection and dependency between BGP pathlist and
1252      IGP pathlist.

[] The last two bullets are a characteristic of the implementation.



1254 7.4. Incremental Deployment

1256   As soon as one router supports BGP-PIC solution, it benefits from
1257   all its benefits (most notably convergence that does not depend in
1258   the number of prefixes) without any requirement for other routers to
1259   support BGP-PIC.

[nit] Reduncant text: "it benefits from all its benefits"


[major] The assertion above is true.  However, (in the extreme) a single
router supporting BGP-PIC doesn't necessarily translate in to better
overall convergence in the network (it depends on where the router is, the
failure, etc.).  Also, having a mix of routers could result in transient
micro loops as the speed of convergence is not the same.



1261 8. Security Considerations

1263   The behavior described in this document is internal functionality
1264   to a router that result in significant improvement to convergence
1265   time as well as reduction in CPU and memory used by FIB while not
1266   showing change in basic routing and forwarding functionality. As
1267   such no additional security risk is introduced by using the
1268   mechanisms proposed in this document.

[] I can't think of anything else to say here, but you did not show (nor do
I need you to) "significant...reduction in CPU and memory used".  It sounds
more like a marketing statement. :-(



1270 9. IANA Considerations

1272   No requirements for IANA

[major] s/.../This document has no IANA actions.



1274 10. Conclusions

[] This section is not needed, it contains redundant information.  Please
remove it.



...
1283 11. References

[major] The RFC Editor requires that the OID be listed for all references
[1].  Please update the references -- here's an example of what they should
look like [2]:

   [RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A Border
              Gateway Protocol 4 (BGP-4)", RFC 4271, DOI 10.17487/RFC4271,
              January 2006, <https://www.rfc-editor.org/info/rfc4271>.

[1] https://www.rfc-editor.org/styleguide/part2/
[2] https://www.rfc-editor.org/refs/ref4271.txt



1285 11.1. Normative References
...
1293   [3]   E. Rosen, " Carrying Label Information in BGP-4", RFC 8277,
1294         October 2017

[minor] This reference can be Informative.



1296   [4]   Andersson, L., Minei, I., and B. Thomas, "LDP Specification",
1297         RFC 5036, October 2007

[minor] This reference can be Informative.



1299   [5]   A. Bashandy, C. Filsfils, S. Previdi, B. Decraene, S.
1300         Litkowski, M. Horneffer, R. Shakir, "Segment Routing with MPLS
1301         data plane", RFC 8660, December 2019

[minor] This reference can be Informative.



...
1377 Appendix A.                 Perspective
...
1392   o  BGP Convergence per BGP destination ~ 200usec conservative,

1394                                          ~ 100usec optimistic

[minor] "BGP Convergence"

The explanation below talks about recalculating the best route -- which is
different than convergence (which includes sending BGP Updates, etc.).

[EoR -18]