[bess] [Shepherding AD review] review of draft-ietf-bess-evpn-ipvpn-interworking-14

["Gunter van de Velde (Nokia)" <[e-mail]>]

Hi Authors, WG,

# Gunter Van de Velde, RTG AD, comments for draft-ietf-bess-evpn-ipvpn-interworking-14

# line numbers are rendered from the idnits tool found at https://author-tools.ietf.org/api/idnits?url=https://www.ietf.org/archive/id/draft-ietf-bess-evpn-ipvpn-interworking-14.txt

# There are 7 authors. This likely will cause hickups during iesg ballot process. Consider reducing to maximal 5 key editors/authors or prepare for justification negotiations and potential blocking DISCUSS positions.

# i scanned through the text and made observations while going through the text or suggesting alternative textblobs to aid readability. Before proceeding further requesting an IETF Last-Call and consequently with the IESG, I would appreciate it if you could consider these.

# It was a more complex read as i anticipated and took much longer to digest as i expected. I do believe that it is a good necessary and useful technology, which will be used often and plenty by operators.

# The terminology section seems a bit light, and excludes terms maybe well known in BESS but not that well known with the general audience. (for example PE, CE, EVI, Eth-Tag, etc.). Maybe scan through the document through the lens of a non-routing participant.

# Many thanks for the shepherd writeup from Stephane.

#DETAILED COMMENTS
#==================


21	Abstract

23	   Ethernet Virtual Private Network (EVPN) is used as a unified control
24	   plane for tenant network intra and inter-subnet forwarding.  When a
25	   tenant network spans not only EVPN domains but also domains where BGP
26	   VPN-IP or IP families provide inter-subnet forwarding, there is a
27	   need to specify the interworking aspects between BGP domains of type
28	   EVPN, VPN-IP and IP, so that the end-to-end tenant connectivity can
29	   be accomplished.  This document specifies how EVPN interworks with
30	   VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families for inter-subnet
31	   forwarding.  The document also addresses the interconnect of EVPN
32	   domains for Inter-Subnet Forwarding routes.  In addition, this
33	   specification defines a new BGP Path Attribute called D-PATH (Domain
34	   PATH) that protects gateways against control plane loops.  D-PATH
35	   modifies the BGP best path selection for multiprotocol BGP routes of
36	   SAFI 128 and EVPN IP Prefix routes, and therefore this document
37	   updates the BGP best path selection specification, but only for IPVPN
38	   and EVPN families.

GV> 

"
This document defines procedures for interworking between Ethernet Virtual Private Network (EVPN) and IP Virtual Private Network (IPVPN) networks to support the delivery of Layer 2 and Layer 3 services across heterogeneous VPN domains. It specifies mechanisms for the exchange of reachability information and the seamless forwarding of traffic between EVPN and IPVPN Provider Edge (PE) devices. These interworking procedures enable scenarios where customer endpoints connected to EVPN PEs can communicate with endpoints connected to IPVPN PEs, and vice versa. The solution leverages existing BGP-based control plane mechanisms and is applicable to both MPLS and Segment Routing data planes.

In addition, this document defines a new BGP Path Attribute, referred to as D-PATH (Domain PATH), which provides loop prevention for gateway nodes by protecting against control plane loops. The introduction of D-PATH modifies the BGP best path selection process for Multiprotocol BGP routes of SAFI 128 (IPVPN) and EVPN IP Prefix routes. As a result, this specification updates the BGP best path selection procedure, but only in the context of IPVPN and EVPN route families.
"

96	1.  Introduction and Problem Statement

98	   EVPN is used as a unified control plane for tenant network intra and
99	   inter-subnet forwarding.  When a tenant network spans not only EVPN
100	   domains but also domains where BGP VPN-IP or IP families provide
101	   inter-subnet forwarding, there is a need to specify the interworking
102	   aspects between the different families, so that the end-to-end tenant
103	   connectivity can be accomplished.  This document specifies how EVPN
104	   should interwork with VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families
105	   for inter-subnet forwarding.  The document also addresses the
106	   interconnect of an EVPN domain to another EVPN domain for Inter-
107	   Subnet Forwarding routes.  In addition, this specification defines a
108	   new BGP Path Attribute called D-PATH (Domain PATH) that protects
109	   gateways against control plane loops.  Loops are created when two (or
110	   more) redundant gateway PEs interconnect two domains and exchange
111	   inter-subnet forwarding routes.  For instance, if PE1 and PE2 are
112	   redundant gateway PEs interconnecting an IPVPN and an EVPN domain,
113	   gateway PE1 receives a VPN-IP route to prefix P and propagates the
114	   route into an EVPN IP Prefix to P.  If gateway PE2 receives the EVPN
115	   IP Prefix route, it cannot propagate the route back to the IPVPN
116	   domain, or it would create a loop for prefix P.
117
118	   D-PATH modifies the BGP best path selection for multiprotocol BGP
119	   routes of SAFI 128 and EVPN IP Prefix routes, and therefore this
120	   document updates the BGP best path selection procedures in [RFC4271]
121	   for IPVPN and EVPN families.
121
123	   EVPN supports the advertisement of IPv4 or IPv6 prefixes in two
124	   different route types:
125
126	   *  Route Type 2 - EVPN MAC/IP Advertisement route (only for /32 and
127	      /128 host routes), as described by [RFC9135].
128
129	   *  Route Type 5 - EVPN IP Prefix route, as described by [RFC9136].
130
131	   When interworking with other BGP address families (AFIs/SAFIs) for
132	   inter-subnet forwarding, the IP prefixes in those two EVPN route
133	   types must be propagated to other domains using different SAFIs.
134	   Some aspects of that propagation must be clarified.  Examples of
135	   these aspects or procedures across BGP families are: route selection,
136	   loop prevention or BGP Path attribute propagation.  The Interworking
137	   PE concepts are defined in Section 3, and the rest of the document
138	   describes the interaction between Interworking PEs and other PEs for
139	   end-to-end inter-subnet forwarding.
140
141	   The interworking procedures in this document always require creating
142	   an IP-VRF on the interworking PE.  When connecting different domains,
143	   the interworking PE follows these steps: it receives routes from one
144	   domain (along with that domain's encapsulation parameters), installs
145	   them in the IP-VRF route table, and then reoriginates the routes with
146	   the encapsulation parameters of the adjacent domain before
147	   advertising them.  This reorigination process ensures that the
148	   procedures remain independent of the specific transport tunnels used
149	   in each domain, as it functions as a service interworking solution.

GV> 

"
EVPN is used as a unified BGP control plane to support both intra-subnet and inter-subnet forwarding for tenant networks. In deployments where a tenant network spans multiple domains, some of which use EVPN, and others which rely on BGP VPN-IPv4/VPN-IPv6 or IPv4/IPv6 address families for inter-subnet forwarding, it becomes necessary to define interworking procedures to enable seamless end-to-end tenant connectivity across these heterogeneous domains.

This document specifies procedures for interworking between EVPN and other BGP address families, including VPN-IPv4, VPN-IPv6, IPv4, and IPv6, specifically for the purpose of inter-subnet forwarding. It also addresses the interconnection of one EVPN domain with another, focusing on the propagation and handling of EVPN inter-subnet forwarding routes across such domains.

To support loop prevention in scenarios where redundant gateway Provider Edges (PEs) interconnect distinct routing domains, this specification introduces a new BGP Path Attribute called the Domain Path (D-PATH). In topologies where multiple gateways connect EVPN and IPVPN domains, control plane loops may occur if routes are redistributed between domains without proper safeguards. For example, if gateway PE1 imports a VPN-IP route for a given prefix and redistributes it as an EVPN IP Prefix route into the EVPN domain, and a second gateway PE (PE2) receives this EVPN route and re-advertises it back into the IPVPN domain, a loop may form. The D-PATH attribute is designed to prevent such scenarios by providing domain-level loop detection and avoidance.

The D-PATH attribute alters the BGP best path selection logic for Multiprotocol BGP routes of SAFI 128 (VPN-IPv4/IPv6) and for EVPN IP Prefix routes. Accordingly, this document updates the BGP best path selection procedures specified in [RFC4271], but only in the context of IPVPN and EVPN families.

EVPN supports the advertisement of IP prefixes through two route types:
* Route Type 2: EVPN MAC/IP Advertisement route, as defined in [RFC9135], supporting host routes (i.e., /32 or /128).
* Route Type 5: EVPN IP Prefix route, as defined in [RFC9136].

When interworking with other BGP address families for inter-subnet forwarding, the IP prefixes conveyed in these EVPN route types must be propagated into corresponding address families (e.g., VPN-IP), and vice versa. Several aspects of this propagation require clarified procedures, including route selection, loop prevention, and BGP Path Attribute handling across AFI/SAFI boundaries.

This document defines the concept of an Interworking PE, which is responsible for bridging different routing domains. An Interworking PE implements the following behavior: it imports routes from one domain (along with the domain-specific encapsulation parameters), installs them in an IP-VRF, and reoriginates the routes with the encapsulation attributes suitable for the adjacent domain before advertisement. This reorigination process enables the solution to operate independently of the transport encapsulation mechanisms used within each domain and serves as a service interworking function.

The procedures defined herein ensure that tenant inter-subnet connectivity can be maintained across a mix of EVPN and non-EVPN domains, while preventing routing loops and maintaining protocol consistency across BGP address families.
"

195	      network.  The ISF SAFIs are 1 (including IPv4 and IPv6 AFIs), 128

GV> is this limited to AFI 1 and 2 ? Maybe say that instead of leaving it open ended for other already existing AFIs or not yet existing AFIs (https://www.iana.org/assignments/address-family-numbers/address-family-numbers.xhtml )

210	   *  MAC-VRF: a MAC Virtual Routing and Forwarding table, as defined in
211	      [RFC7432].  It is also the instantiation of an EVI (EVPN Instance)
212	      in a PE.  Route Distinguisher and Route Target(s) are required
213	      properties and they are normally different than the ones defined
214	      in the associated IP-VRF (if there is an associated IP-VRF linked
215	      to a Bridge Table of the MAC-VRF, via IRB interface).

GV>

"
MAC-VRF: A MAC Virtual Routing and Forwarding table, as defined in [RFC7432]. A MAC-VRF represents the instantiation of an EVPN Instance (EVI) on a PE device. Each MAC-VRF is associated with a unique Route Distinguisher (RD) and one or more Route Targets (RTs), which are required attributes for its operation. These RD and RT values are typically distinct from those used by any associated IP-VRF, when such an IP-VRF is linked to the MAC-VRF through a Bridge Table via an Integrated Routing and Bridging (IRB) interface.
"

217	   *  BT: a Bridge Table, as defined in [RFC7432].  A BT is the
218	      instantiation of a Broadcast Domain in a PE.  When there is a
219	      single Broadcast Domain in a given EVI, the MAC-VRF in each PE
220	      will contain a single BT.  When there are multiple BTs within the
221	      same MAC-VRF, each BT is associated to a different Ethernet Tag.
222	      The EVPN routes specific to a BT, will indicate which Ethernet Tag
223	      the route corresponds to.

GV>

"
BT: A Bridge Table, as defined in [RFC7432], represents the instantiation of a Broadcast Domain on a PE device. When an EVI contains a single Broadcast Domain, the associated MAC-VRF on each PE includes a single BT. In cases where multiple Broadcast Domains exist within the same MAC-VRF, each BT is associated with a distinct Ethernet Tag. EVPN routes specific to a given BT include the corresponding Ethernet Tag to indicate the Broadcast Domain to which the route pertains.
"

231	      (where the VLAN tags can be individual c-tags, s-tags or ranges of
232	      both).

GV> Would references for s-tag, c-tag make sense?

241	   *  MPLS/NVO tunnel: It refers to a tunnel that can be MPLS or NVO-
242	      based (Network Virtualization Overlays) and it is used by MAC-VRFs
243	      and IP-VRFs.  Irrespective of the type, the tunnel may carry an
244	      Ethernet or an IP payload.  MAC-VRFs can only use tunnels with
245	      Ethernet payloads (setup by EVPN), whereas IP-VRFs can use tunnels
246	      with Ethernet (setup by EVPN) or IP payloads (setup by EVPN or
247	      IPVPN).  IPVPN-only PEs have IP-VRFs but they cannot send or
248	      receive traffic on tunnels with Ethernet payloads.

GV> 

"
MPLS/NVO Tunnel: A tunnel that may be based on either MPLS or a Network Virtualization Overlay (NVO) technology. Such tunnels are utilized by both MAC-VRFs and IP-VRFs. Regardless of the underlying tunneling technology, the tunnel may carry either Ethernet or IP payloads. MAC-VRFs are restricted to using tunnels that carry Ethernet payloads, which are typically established via EVPN signaling. In contrast, IP-VRFs may utilize tunnels carrying Ethernet payloads (signaled via EVPN) or IP payloads (signaled via EVPN or IPVPN mechanisms). IPVPN-only PE devices support IP-VRFs but do not support sending or receiving traffic over tunnels carrying Ethernet payloads.
"

GV> in the above, can a reference pointer be added for the IP-VRF carying EThernet payloads, as that sound conter intuitive at first glance

250	      Example: Figure 1 shows an MPLS/NVO tunnel that is used to
251	      transport Ethernet frames to/from MAC-VRF1.  The PE determines the
252	      MAC-VRF and BT the packets belong to based on the EVPN label (MPLS
253	      or VNI).  Figure 1 also shows two MPLS/NVO tunnels being used by
254	      IP-VRF1, one carrying Ethernet frames and the other one carrying
255	      IP packets.

GV>

"
Example: Figure 1 illustrates the use of an MPLS or NVO-based tunnel to transport Ethernet frames associated with MAC-VRF1. The PE device identifies the corresponding MAC-VRF and BT based on the EVPN label, either an MPLS label or a Virtual Network Identifier (VNI), depending on the encapsulation type.

Additionally, Figure 1 shows two distinct MPLS/NVO tunnels used by IP-VRF1: one tunnel transports Ethernet frames, while the other carries IP packets. This demonstrates that IP-VRFs may concurrently utilize multiple tunnel types, depending on the payload and the signaling mechanism (EVPN or IPVPN).

"

263	   *  Domain: Two PEs are in the same domain if they are attached to the
264	      same tenant and the packets between them do not require a data
265	      path IP lookup (in the tenant space) in any intermediate router.
266	      A gateway PE is always configured with multiple DOMAIN-IDs.  The
267	      domain boundaries are not limited to an Autonomous System or an
268	      IGP instance.  The PEs in a domain can all be part of the same or
269	      different Autonomous System, and an Autonomous System can also
270	      contain multiple domains.

GV> What is an intermediate router? I understand what this blob is trying to describe. Could this section be clarified for those beyond EVPN or routing area?

340	   *  Interworking PE: a PE that may advertise a given prefix with an
341	      EVPN ISF route (EVPN MAC/IP Advertisement route or EVPN IP Prefix
342	      route) and/or an IPVPN ISF route and/or a BGP IP ISF route.  An
343	      interworking PE has one IP-VRF per tenant, and zero, one or
344	      multiple MAC-VRFs per tenant.  Each MAC-VRF may contain one or
345	      more BTs, where each BT may be attached to that IP-VRF via IRB.
346	      There are two types of Interworking PEs: composite PEs and gateway
347	      PEs.  Both PE functions can be independently implemented per
348	      tenant and they may both be implemented for the same tenant.

GV>

"
Interworking PE: A PE device that is capable of advertising a given IP prefix using one or more of the following route types: an EVPN Inter-Subnet Forwarding (ISF) route, either an EVPN MAC/IP Advertisement route or an EVPN IP Prefix route-an IPVPN ISF route, or a BGP IP ISF route. An Interworking PE maintains a single IP-VRF per tenant and zero, one, or more MAC-VRFs per tenant. Each MAC-VRF may include one or more Bridge Tables (BTs), and each BT may be associated with the tenant's IP-VRF via an Integrated Routing and Bridging (IRB) interface.

  There are two types of Interworking PEs:

  1. Composite PE
  2. Gateway PE

  These two functions may be implemented independently on a per-tenant basis and may also coexist for the same tenant on a single PE.
"

355	   *  Composite PE: an interworking PE that is attached to a composite
356	      domain and advertises a given prefix to an IPVPN peer with an
357	      IPVPN ISF route, to an EVPN peer with an EVPN ISF route, and to a
358	      route reflector with both an IPVPN and EVPN ISF route.  A
359	      composite PE performs the procedures of Section 7.

GV>

"
Composite PE: An Interworking PE device that is connected to a composite domain and is capable of advertising a given prefix to multiple types of peers using appropriate route types. Specifically, a Composite PE advertises the prefix to an IPVPN peer using an IPVPN ISF route, to an EVPN peer using an EVPN ISF route, and to a route reflector using both IPVPN and EVPN ISF routes. A Composite PE implements the procedures defined in Section 7.
"

380	   *  Gateway PE: an interworking PE that is attached to two (or more)
381	      domains, each either regular or composite.  The Gateway PE can
382	      have IBGP and/or EBGP peers on the domains that is connecting.
383	      Based on configuration, the Gateway PE does one of the following:
384
385	      -  Propagates ISF routes of the same ISF SAFI, i.e., BGP IP, IPVPN
386	         or EVPN, between the two domains.
387
388	      -  Propagates an ISF route received with an ISF SAFI to a domain
389	         that uses a different ISF SAFI.  E.g., it propagates a received
390	         EVPN ISF route as an IPVPN ISF route in the other domain and
391	         vice versa.  A gateway PE performs the procedures of Section 8.
391
393	      A gateway PE is always configured with multiple DOMAIN-IDs.  The
394	      DOMAIN-ID is encoded in the Domain Path Attribute (D-PATH), and
395	      advertised along with ISF SAFI routes.  Section 4 describes the
396	      D-PATH attribute.

GV>

"
Gateway PE: An Interworking PE device that connects two or more distinct domains, where each domain may be either a regular domain or a composite domain. A Gateway PE may establish either IBGP or EBGP sessions with peers in the connected domains.

  Depending on its configuration, the Gateway PE performs one of the following functions:

  * Propagates ISF routes using the same ISF SAFI, such as BGP IP, IPVPN, or EVPN, between the connected domains.

  * Translates and propagates an ISF route received with one ISF SAFI to a domain that uses a different ISF SAFI. For example, a received EVPN ISF route may be propagated as an IPVPN ISF route, and vice versa.

  * The Gateway PE follows the procedures defined in Section 8.

A Gateway PE is always configured with multiple DOMAIN-IDs. These DOMAIN-IDs are encoded in the D-PATH and are included in ISF SAFI route advertisements. The structure and behavior of the D-PATH attribute are described in Section 4.
"

412	   *  Composite/Gateway PE: an interworking PE that is both a composite
413	      PE and a gateway PE that is attached to two domains, one regular
414	      and one composite, and which does the following:

416	      -  Propagates an ISF route from the regular domain into the
417	         composite domain.  Within the composite domain it acts as a
418	         composite PE.

420	      -  Propagates an ISF route from the composite domain into the
421	         regular domain.  Within the regular domain it is propagated as
422	         an ISF route using the ISF SAFI for that domain.

424	      This is particularly useful when a tenant network uses multiple
425	      ISF SAFIs (BGP IP, IPVPN and EVPN domains) and any-to-any
426	      connectivity is required.  In this case end-to-end control plane
427	      consistency, when possible, is desired.

GV> 

"
Composite/Gateway PE: An Interworking PE device that simultaneously performs the functions of both a Composite PE and a Gateway PE. This type of PE is connected to two domains: one regular domain and one composite domain. It operates as follows:

* Propagates an ISF route received from the regular domain into the composite domain. Within the composite domain, it performs the behavior of a Composite PE.

* Propagates an ISF route received from the composite domain into the regular domain. In the regular domain, the route is advertised using the ISF SAFI applicable to that domain.

This functionality is particularly useful in scenarios where a tenant network spans multiple domains using different ISF SAFIs (e.g., BGP IP, IPVPN, and EVPN), and where any-to-any tenant connectivity is required. In such deployments, maintaining consistent end-to-end control plane behavior across domains is desirable when feasible.
"

GV> inthe above maybe better to use the real SAFI names as allocated by IANA instead of praphrasing them, or make it clear that the SAFIs are paraphrased

431	   The BGP Domain Path (D-PATH) attribute is an optional and transitive
432	   BGP path attribute.

GV> D-PATH is expanded prior to this usage. It is even expanded in the paragraph title. Maybe keep it by a single expansion. 

434	   Similar to AS_PATH, D-PATH is composed of a sequence of Domain

GV> oddball question. Why is AS_Path using underscore, and D-PATH using dash? Would using a underscore for noth not be more consistant?


440	   Octets
441	   0               1                    8                         n
442	   +---------------+----------------//--+----//-------------------+
443	   |Domain Segment |   Last Domain      |        Domain of Origin |
444	   |    Length     |                    |                         |
445	   +---------------+----------------//--+----//-------------------+

GV> The "Last Domain" and the "Domain of Origin" is not listed in the next paragraohs. Is that intentional.
Also the fields usage of camel found within Figure 6 is not consistently reflected back in the next table describing the fields.

GV> The field explanations do not have a easy to spot consistent field length. I personaly find it convinient to add something like for example (length: 1 octet) 


464	   *  The domain segment length field is a 1-octet field, containing the
465	      number of domains in the segment.

GV>

"
* Domain Segment Length (length: 1-octet): containing the number of domains in the segment.
"

474	      D-PATH in ISF routes.  The Local Administrator sub-field is any
475	      local 2-octet value, and its allocation or configuration is a
476	      local implementation matter.  The representation of the Global

GV> Is this to be used as a nounce? maybe use that term if that is the usage

476	      local implementation matter.  The representation of the Global
477	      Administrator and Local Administrator values as opaque unsigned
478	      integers is RECOMMENDED.

GV> What is meant with "representation"? is that the correct wording?

480	   *  ISF_SAFI_TYPE is a 1-octet field that indicates the Inter-Subnet
481	      Forwarding SAFI type in which a route was received, before the
482	      route is re-exported into a different domain.  The ISF_SAFI_TYPE

GV received by whom? and re-exported by whom?

499	   The BGP D-PATH attribute is supported on ISF routes of type IPVPN and
500	   EVPN and MUST NOT be advertised along with routes different from
501	   IPVPN and EVPN routes.  By default, the BGP D-PATH attribute is not

GV> What happens if it is appended/received attached to route types that are not suported?
GV> WHat happens if two or more of these are attached? Can this ever happen?

505	   a.  Identifies the sequence of domains, each identified by a <DOMAIN-
506	       ID:ISF_SAFI_TYPE> through which a given ISF route of type IPVPN
507	       or EVPN has passed.

GV> The Figure 6 only shows the Last Domain and the Domain of origin. I suspect there are a bunch of these inbetween.
Maybe the Figure can be enhanced to more clearly represent the intended structure? (similar as AS_PATH)

509	       *  This attribute list MAY contain one or more segments.  Each
510	          segment's Domain Segment Length MUST be equal or greater than
511	          one.

GV> The text here assumes a single NLRI may have multiple of these D-PATH lists, is that correct?
Is that intended? Is there a usecase for that? if only a single D-PATH is ever expected, then maybe add exception handling in case it differs

GV> Any specific handling if the length is "0"?

522	       *  As an example, an ISF IPVPN or EVPN route received with a
523	          D-PATH attribute containing a domain segment of {length=2,
524	          <6500:2:IPVPN>,<6500:1:EVPN>} indicates that the route was
525	          originated in EVPN domain 6500:1, and propagated into IPVPN
526	          domain 6500:2.

GV> Would {length=2,<6500:1:IPVPN>,<6500:1:EVPN>} be a valid list?

528	       *  In order to minimize the number of segments in the D-PATH
529	          attribute, the local gateway PE prepends its own domain as the
530	          last element of the domain segment.  If the act of prepending
531	          a new domain causes an overflow in the domain segment (i.e.,
532	          more than 255 domains), the local gateway PE MUST prepend a
533	          new segment and prepend its own domain to this new segment.

GV> This reads confusing to me. First it states that a PE prepend of its own domain  may case overflow, and then to fix that it MUST prepend new segment with its own domain. I am not sure how this works as the prpend causes overflow? Does this mean that a 2nd (or 3rd or 4th) list is added?

535	   b.  It is added/modified by a gateway PE when propagating an update

GV> What is "It". WHen reading with the prior header "In addition, D-PATH:" then removing the "It" seems to read better.

549	       *  For instance, in an IP-VRF configured with DOMAIN-IDs 6500:1
550	          for EVPN and 6500:2 for IPVPN, if an EVPN route for prefix P
551	          is received and P installed in the IP-VRF, the IPVPN route for
552	          P that is exported to an IPVPN peer will prepend the domain
553	          <6500:1:EVPN> to the previously received D-PATH attribute.
554	          Likewise, IP-VRF prefixes that are received from IP-VPN, will
555	          be exported to EVPN peers with the domain <6500:2:IPVPN> added
556	          to the segment.

GV>

"
For instance, consider an IP-VRF configured with DOMAIN-IDs 6500:1 for EVPN and 6500:2 for IPVPN. If an EVPN route for prefix P is received and P is installed in the IP-VRF, then the corresponding IPVPN route for P, when exported to an IPVPN peer, will include the domain identifier <6500:1:EVPN> prepended to the existing D-PATH attribute. Similarly, prefixes received in the IP-VRF from an IPVPN peer will be exported to EVPN peers with the domain identifier <6500:2:IPVPN> appended to the D-PATH attribute.
"

563	       *  Within the Domain of Origin, the update does not contain a
564	          D-PATH attribute because the update has not passed through a
565	          gateway PE yet.

GV> I got slightly off track with the term "Domain of Origin" as that is not a well known BGP or BESS naming. It seems implicitly defined what it is on line 518, but maybe add it to the terminology and conventions used section

567	   c.  For a local ISF route, i.e., a configured route or a route
568	       learned from a local attachment circuit, a gateway PE has three
569	       choices:

GV> Maybe clarify that this assumes this in the context of this specific specification? (it may have many other options/choices beyond this context)

571	       1.  It MAY advertise that ISF route without a D-PATH attribute
576	       2.  It MAY advertise that ISF route with a D-PATH attribute into

GV> Maybe replace "It MAY" with a more accurate description not using the word "It". As alternative one could write "The PE MAY advertise ..."

582	       3.  It MAY advertise that ISF route with a D-PATH attribute that
583	           contains a configured domain specific to its local ISF routes
584	           into one or more of its configured domains, in which case the
585	           D-PATH attribute in each copy of the ISF route is initialized
586	           with a ISF_SAFI_TYPE of 0 and the DOMAIN-ID for the local ISF
587	           routes.  This DOMAIN-ID MUST be globally unique and MAY be
588	           shared by two or more gateway PEs.  Although the three
589	           options help detect control plane loops, this option 3 is
590	           RECOMMENDED, since it is the option that provides more
591	           information about the differentiated origin of the route (it
592	           uses a DOMAIN-ID and ISF_SAFI_TYPE that identifies the route
593	           as a local gateway route).

GV> Maybe split up this paragraph for readability and to make it more apparent that the last paragraph is relevant to the three bullet items.

"
3. The PE MAY advertise the ISF route with a D-PATH attribute containing a locally configured domain identifier associated with its local ISF routes into one or more of its configured domains. In this case, the D-PATH attribute in each copy of the ISF route is initialized with an ISF_SAFI_TYPE value of 0 and the DOMAIN-ID representing the local ISF domain. The DOMAIN-ID MUST be globally unique and MAY be shared across multiple gateway PEs.

Although all three options provide mechanisms for detecting control plane loops, this third option is RECOMMENDED, as it conveys additional information about the origin of the route. Specifically, it allows the receiving PE to identify the route as having originated from a local gateway, based on the combination of the DOMAIN-ID and the ISF_SAFI_TYPE value.
"

595	   d.  An ISF IPVPN or EVPN route received by a gateway PE with a D-PATH
596	       attribute that contains one or more of its locally associated
597	       DOMAIN-IDs (for the IP-VRF) is considered to be a looped ISF
598	       route for the purpose of re-exporting the route to the adjacent
599	       domain in a Gateway PE.  The ISF route in this case MUST be
600	       flagged as "looped", MUST NOT be exported, and MAY be installed
601	       in the IP-VRF only in case there is no better route after the
602	       best path selection (Section 6).  The ISF_SAFI_TYPE is irrelevant
603	       for the purpose of loop detection of an ISF route.  In other
604	       words, an ISF route is considered as a looped route if it
605	       contains a D-PATH attribute with at least one DOMAIN-ID matching
606	       a local DOMAIN-ID, irrespective of the ISF_SAFI_TYPE of the
607	       DOMAIN-ID.
608
609	       For instance, in the example of Figure 2, gateway GW1 receives
610	       TS1 prefix in two different ISF routes:
611
612	       *  In an EVPN IP Prefix route with next-hop PE1 and no D-PATH
613	          attribute.
614
615	       *  In a SAFI 128 route with next-hop GW2 and D-PATH = {length=1,
616	          <6500:1:EVPN>}, assuming that DOMAIN-ID for domain 1 is
617	          6500:1.
618
619	       Gateway GW1 flags the SAFI 128 route as "looped" (since 6500:1 is
620	       a local DOMAIN-ID in GW1) and it will not install it in the
621	       tenant IP-VRF, since the route selection process selects the EVPN
622	       IP Prefix route due to a shorter D-PATH attribute (Section 6).
623	       Gateway GW1 identifies the route as "looped" even if the
624	       ISF_SAFI_TYPE value is unknown to GW1, i.e., any value different
625	       from the ones specified in this document).

GV>

"
d. An ISF route of type IPVPN or EVPN received by a Gateway PE that includes a D-PATH attribute containing one or more DOMAIN-ID values locally associated with the corresponding IP-VRF MUST be considered a looped ISF route for the purposes of re-advertisement into adjacent domains. In such cases:

* The ISF route MUST be flagged as "looped".
* The route MUST NOT be re-exported to any other domain.
* The route MAY be installed in the IP-VRF only if it is selected as the best path according to the procedures defined in Section 6.

For the purpose of loop detection, the ISF_SAFI_TYPE value associated with a DOMAIN-ID in the D-PATH attribute is irrelevant. That is, a route is considered looped if it contains at least one DOMAIN-ID that matches any local DOMAIN-ID configured on the Gateway PE, regardless of the ISF_SAFI_TYPE value.

Example: In the scenario illustrated in Figure 2, gateway GW1 receives two ISF routes for the same prefix associated with TS1:

* An EVPN IP Prefix route with a next-hop of PE1, and no D-PATH attribute.
* A SAFI 128 (IPVPN) route with a next-hop of GW2, and a D-PATH attribute containing a single segment: {length=1, <6500:1:EVPN>}, where 6500:1 is assumed to be the DOMAIN-ID for domain 1, which is local to GW1.

Upon receiving the SAFI 128 route, GW1 identifies 6500:1 as a locally configured DOMAIN-ID, and therefore flags the route as "looped". As a result, GW1 does not install this route in the tenant IP-VRF, because the route selection process prefers the EVPN IP Prefix route (due to its shorter D-PATH attribute, as specified in Section 6). Loop detection is applied even if the ISF_SAFI_TYPE value in the D-PATH attribute is unknown to GW1 or does not match any SAFI defined in this specification.
"

627	   e.  A DOMAIN-ID value on a gateway PE MAY be assigned for a peering
628	       domain or MAY be scoped for an individual tenant IP-VRF.
629
630	       *  If allocated for a peering domain, the DOMAIN-ID applies to
631	          all tenant IP-VRFs for that domain.
632
633	       *  If allocated for a specific tenant IP-VRF, the processing of
634	          the received D-PATH and its propagation is in the context of
635	          the IP-VRF DOMAIN-ID.  Route leaking is a use-case where a
636	          per-IP-VRF DOMAIN-ID assignment is necessary.  Suppose
637	          gateways PE1 and PE2 are attached to two different tenant IP-
638	          VRFs, IP-VRF-1 and IP-VRF-2.  ISF SAFI routes advertised by
639	          gateway PE1 for IP-VRF-1 are received on gateway PE2 with
640	          DOMAIN-ID 6500:1.  If the routes are leaked from IP-VRF-1 into
641	          IP-VRF-2 on PE2, and re-advertised back to PE1 in the context
642	          of IP-VRF-2, PE1 will not identify the route as a looped
643	          route.  This is because PE1 processes the route in the context
644	          of IP-VRF-2, where DOMAIN-ID 6500:1 is not a local DOMAIN-ID.

GV>

"
e. A DOMAIN-ID configured on a Gateway PE MAY be assigned at either the peering domain level or scoped individually per tenant IP-VRF.

* When the DOMAIN-ID is allocated at the peering domain level, it SHALL apply to all tenant IP-VRFs associated with that domain.
* When the DOMAIN-ID is allocated for a specific tenant IP-VRF, the processing of received D-PATH attributes and their subsequent propagation SHALL be performed in the context of that IP-VRF's DOMAIN-ID.

A per tenant IP-VRF DOMAIN-ID assignment is particularly useful in scenarios involving route leaking. For example, consider two Gateway PEs, PE1 and PE2, each associated with different tenant IP-VRFs, denoted as IP-VRF-1 and IP-VRF-2, respectively. If PE1 advertises ISF SAFI routes for IP-VRF-1 with a DOMAIN-ID of 6500:1, and these routes are received on PE2 and subsequently leaked from IP-VRF-1 into IP-VRF-2, the re-advertisement of the routes from PE2 back to PE1 in the context of IP-VRF-2 will not be considered looped by PE1. This is because PE1 processes the route in the context of IP-VRF-2, for which DOMAIN-ID 6500:1 is not locally configured.
"

646	   f.  The number of domains in the D-PATH attribute indicates the
647	       number of gateway PEs that the ISF route update has transited.
648	       If one of the transit gateway PEs leaks a given ISF route between
649	       two local IP-VRFs, it MAY prepend a domain with a ISF_SAFI_TYPE
650	       of 0 for the leaked route when the route is exported into an ISF
651	       SAFI.  In that case, the number of domains in the D-PATH
652	       attribute indicates the number of tenant IP-VRFs that the ISF
653	       route update has transited.

GV>

"
f. The number of domains encoded in the D-PATH attribute reflects the number of Gateway PEs that the corresponding ISF route update has traversed. If a transit Gateway PE performs route leaking between two local tenant IP-VRFs, it MAY prepend a domain segment to the D-PATH attribute with an ISF_SAFI_TYPE value of 0 when exporting the leaked route into an ISF SAFI. In such cases, the total number of domain entries in the D-PATH attribute represents the number of tenant IP-VRFs through which the ISF route update has propagated.
"

655	   g.  The following error-handling rules apply to the D-PATH attribute:
656
657	       1.  A received D-PATH attribute MUST be considered malformed if
658	           it contains a malformed Domain Segment.
659
660	       2.  A Domain Segment MUST be considered malformed in any of the
661	           following cases:
662
663	           *  The length of the Domain Segment is longer than the D-PATH
664	              attribute that contains it.
665
666	           *  After the last successfully parsed Domain Segment there
667	              are less than eight octets remaining.
668
669	           *  The D-PATH attribute length is less than 8 octets.
670
671	           *  For each contained Domain Segment, the Domain Segment
672	              length is one octet, containing the number of Domains in
673	              this segment, each of which are 7 octets in length.  If
674	              the total length of the Domain Segment in octets (1 + 7 *
675	              number of Domains) exceeds the remaining length of the
676	              D-PATH attribute, the Domain Segment is malformed.
677
678	       3.  A PE receiving an UPDATE message with a malformed D-PATH
679	           attribute SHALL apply "treat-as-withdraw" [RFC7606].
680
681	       4.  Domains in the D-PATH attribute with unknown ISF_SAFI_TYPE
682	           values are accepted and not considered an error.
683
684	       5.  The D-PATH Path Attribute MUST NOT occur more than once in
685	           the BGP UPDATE's Path Attributes.  If the D-PATH Path
686	           Attribute appears more than once in an UPDATE message, then
687	           all the occurrences of the attribute other than the first one
688	           are discarded and the UPDATE message will continue to be
689	           processed, as per [RFC7606].
690
691	       6.  D-PATH can be advertised with SAFI 128 and EVPN routes and
692	           MUST NOT be sent with any other AFI/SAFIs.  If D-PATH is
693	           received along with routes of AFI/SAFI different from the
694	           IPVPN and EVPN families, the behavior treat-as-withdraw is
695	           applied [RFC7606].

GV>

"
g. The following error-handling procedures apply to the D-PATH Path Attribute:

1. A received D-PATH attribute MUST be considered malformed if it contains a malformed Domain Segment.

2. A Domain Segment MUST be considered malformed under any of the following conditions:

   * The length of the Domain Segment exceeds the remaining length of the enclosing D-PATH attribute.
   * Fewer than eight octets remain after the last successfully parsed Domain Segment.
   * The total length of the D-PATH attribute is less than eight octets.
   * Each Domain Segment consists of a one-octet length field indicating the number of Domains in the segment, with each Domain encoded in seven octets. If the total length of the Domain Segment (i.e., 1 + 7 × number of Domains) exceeds the remaining length of the D-PATH attribute, the Domain Segment is considered malformed.

3. A BGP speaker receiving an UPDATE message containing a malformed D-PATH attribute SHALL apply the "treat-as-withdraw" procedure, as specified in [RFC7606].

4. Domains within the D-PATH attribute that contain unrecognized ISF_SAFI_TYPE values MAY be accepted and MUST NOT be considered an error.

5. The D-PATH Path Attribute MUST NOT appear more than once in the Path Attributes of a given BGP UPDATE message. If multiple instances of the D-PATH attribute are present, all instances other than the first MUST be discarded, and the UPDATE message MUST continue to be processed, as per [RFC7606].

6. The D-PATH Path Attribute MAY be included only in UPDATE messages that carry routes of SAFI 128 (IPVPN) or EVPN. It MUST NOT be included with any other AFI/SAFI combinations. If a D-PATH attribute is received in an UPDATE message associated with an unsupported AFI/SAFI, the "treat-as-withdraw" procedure MUST be applied, in accordance with [RFC7606].
"

697	   h.  The use of D-PATH is restricted to "walled garden" Virtual
698	       Private Networks, and the operator MUST NOT turn on the
699	       generation of D-PATH along with IPVPN and/or EVPN routes if there
700	       are CEs attached to a PE (of any domain in the Virtual Private
701	       Network) that are connected to the Internet.  In addition, a
702	       Gateway PE MUST support the removal of the D-PATH attribute on
703	       import and on export, based on configuration.

GV> 

"
h. The use of the D-PATH attribute is restricted to "walled garden" Virtual Private Network (VPN) deployments. An operator MUST NOT enable the generation of D-PATH attributes in conjunction with IPVPN and/or EVPN routes if any CE devices connected to a Provider Edge (PE) device, belonging to any domain within the VPN, is also connected to the public Internet.

Furthermore, a Gateway PE MUST support the ability to remove the D-PATH attribute upon route import and export, as determined by local configuration.
"

705	5.  BGP Path Attribute Propagation across Domains
706
707	   Based on its configuration, a gateway PE is required to propagate an
708	   ISF route between two domains that use the same or different ISF
709	   SAFI.  This requires a definition of what a gateway PE has to do with
710	   BGP Path Attributes attached to the ISF route that the gateway PE is
711	   propagating.  This section specifies the BGP Path Attribute
712	   propagation modes that a gateway PE may follow when receives an ISF
713	   route with ISF SAFI-x, installs the route in the IP-VRF and exports
714	   the ISF route into ISF SAFI-y.  ISF SAFI-x and SAFI-y values MAY be
715	   the same values.

GV>

"
5. BGP Path Attribute Propagation Across Domains

A Gateway PE device, depending on its local configuration, is required to propagate an ISF route between two domains that utilize either the same or different ISF SAFIs. This requires defining how a Gateway PE handles the BGP Path Attributes associated with the ISF route during such propagation.

This section specifies the BGP Path Attribute propagation behaviors that a Gateway PE MAY apply when it receives an ISF route with ISF SAFI x, installs the route into the relevant IP-VRF, and subsequently re-advertises the route as an ISF route using ISF SAFI y. The values of ISF SAFI x and SAFI y MAY be the same or different.
"

717	5.1.  No-Propagation-Mode
718
719	   This is the default mode of operation for gateway PEs that re-export
720	   ISF routes from a domain into another domain.  In this mode, the
721	   gateway PE will simply re-initialize the BGP Path Attributes when
722	   propagating an ISF route, as it would for direct or local IP
723	   prefixes.  This model may be enough in those use-cases where, e.g.,
724	   the EVPN domain is considered an "abstracted" CE and remote IPVPN/IP
725	   PEs don't need to consider the original EVPN Attributes for path
726	   computations.
719
728	   Since this mode of operation does not propagate the D-PATH attribute
729	   either, redundant gateway PEs are exposed to routing loops.  Those
730	   loops may be resolved by policies and the use of other attributes,
731	   such as the Route Origin extended community [RFC4360], however not
732	   all the loop situations may be identified.

GV>

"
5.1. No-Propagation Mode

The No-Propagation Mode is the default operational mode for Gateway PEs when re-exporting ISF routes from one domain into another. In this mode, the Gateway PE re-initializes the BGP Path Attributes during the propagation of an ISF route, treating it in the same manner as a directly connected or locally originated IP prefix.

This mode is suitable for deployment scenarios where the source domain-for example, an EVPN domain is "abstracted" and treated as a virtual CE, and where remote IPVPN or IP-based PEs do not require the original EVPN-specific BGP Path Attributes for path selection or policy evaluation.

It is important to note that, in No-Propagation Mode, the D-PATH attribute is not propagated. As a result, redundant Gateway PEs may be susceptible to routing loops. While such loops may be mitigated using routing policies or additional attributes, such as the Route Origin extended community [RFC4360], this approach does not guarantee detection or prevention of all potential loop scenarios.
"

734	5.2.  Uniform-Propagation-Mode
735
736	   In this mode, the gateway PE simply keeps accumulating or mapping
737	   certain key commonly used BGP Path Attributes when propagating an ISF
738	   route.  This mode is typically used in networks where EVPN and IPVPN
739	   SAFIs are used seamlessly to distribute IP prefixes.
740
741	   The following rules MUST be observed by the gateway PE when
742	   propagating BGP Path Attributes:
743
744	   1.  The gateway PE imports an ISF route in the IP-VRF and stores the
745	       original Path Attributes.  The following set of Path Attributes
746	       SHOULD be propagated by the gateway PE when advertising the ISF
747	       route to a different domain (other BGP Path Attributes SHOULD NOT
748	       be propagated):
749
750	       *  AS_PATH
751
752	       *  D-PATH, only when advertising SAFI 128 and EVPN routes.
753
754	       *  IBGP-only Path Attributes (when advertising to IBGP peers):
755	          LOCAL_PREF, ORIGINATOR_ID, CLUSTER_ID
756
757	       *  MED
758
759	       *  AIGP
760
761	       *  Communities, Extended Communities, Large Communities and Wide
762	          Communities [I-D.ietf-idr-wide-bgp-communities], except in the
763	          exception cases detailed in point 4 of this section.
764
765	   2.  When propagating an ISF route to a different IBGP peer, the
766	       gateway PE SHOULD keep the AS_PATH of the originating ISF route
767	       and add it to the destination ISF SAFI without any modification.
768	       When re-advertising to an EBGP peer, the gateway PE SHOULD keep
769	       the AS_PATH of the originating ISF route and prepend the IP-VRF's
770	       AS before sending the route.
771
772	   3.  When propagating an ISF route to IBGP peers, the gateway PE
773	       SHOULD keep the IBGP-only Path Attributes from the originating
774	       route to the re-advertised route.  As the ISF route is re-
775	       originated, the route reflector function [RFC4456] is not
776	       required on the gateway PE.
777
778	   4.  As discussed in point 1, Communities, Extended Communities, Large
779	       Communities and Wide Communities SHOULD be preserved from the
780	       originating ISF route by the gateway PE.  Exceptions of Extended
781	       Communities that SHOULD NOT be propagated are:
782
783	       a.  BGP Encapsulation extended communities [RFC9012].
784
785	       b.  Route Target extended communities.  Route Targets are always
786	           initialized when readvertising an ISF route into a different
787	           domain, i.e., they are not propagated.  The initialized Route
788	           Target in the re-advertised ISF route may or may not have the
789	           same value as the Route Target of the originating ISF route.
790
791	       c.  All the extended communities of type EVPN.
792
793	       The gateway PE SHOULD NOT copy the above extended communities
794	       from the originating ISF route to the re-advertised ISF route.
795
796	   5.  For a given ISF route, only the BGP Path Attributes of the best
797	       path can be propagated to another ISF route.  If multiple paths
798	       are received for the same route in an ISF SAFI, the BGP best path
799	       selection will determine what the best path is, and therefore the
800	       set of Path Attributes to be propagated.  Even if Equal Cost
801	       Multi-Path (ECMP) is enabled on the IP-VRF by policy, only the
802	       BGP Path Attributes of the selected best path are propagated.

GV>

"
5.2. Uniform Propagation Mode

In Uniform Propagation Mode, the Gateway PE retains and propagates a consistent set of commonly used BGP Path Attributes when re-advertising an ISF route between domains. This mode is typically employed in deployments where IP prefixes are seamlessly distributed using both EVPN and IPVPN SAFIs.

The following normative behavior MUST be followed by a Gateway PE operating in Uniform Propagation Mode:

1. Upon receiving an ISF route, the Gateway PE MUST import the route into the associated IP-VRF and store the original BGP Path Attributes. When advertising the route into a different domain, the Gateway PE SHOULD propagate only the following set of attributes. All other Path Attributes SHOULD NOT be propagated:

   * AS_PATH
   * D-PATH (only when advertising IPVPN [SAFI 128] or EVPN routes)
   * IBGP-only attributes (when advertising to IBGP peers): LOCAL_PREF, ORIGINATOR_ID, CLUSTER_ID
   * MULTI_EXIT_DISC (MED)
   * AIGP
   * COMMUNITY, EXTENDED_COMMUNITY, LARGE_COMMUNITY, and WIDE_COMMUNITY (as defined in [I-D.ietf-idr-wide-bgp-communities]), except where explicitly excluded in Item 4 below.

2. When re-advertising an ISF route to an IBGP peer, the Gateway PE SHOULD preserve the AS_PATH of the original ISF route without modification. When re-advertising to an EBGP peer, the Gateway PE SHOULD prepend the IP-VRF's ASN to the preserved AS_PATH.

3. When propagating an ISF route to IBGP peers, the Gateway PE SHOULD retain IBGP-only attributes (e.g., LOCAL_PREF, ORIGINATOR_ID, CLUSTER_ID) from the original ISF route. As the route is re-originated, the Gateway PE is not required to perform the route reflector function described in [RFC4456].

4. As stated in Item 1, the Gateway PE SHOULD preserve the Community, Extended Community, Large Community and Wide Community attributes from the original ISF route. However, the following Extended Community types SHOULD NOT be propagated:

   a. BGP Encapsulation Extended Communities, as defined in [RFC9012].
   b. Route Target Extended Communities. Route Targets MUST NOT be propagated and MUST be re-initialized when re-advertising the ISF route into a different domain. The re-initialized Route Target value MAY or MAY NOT match the value used in the original route.
   c. All EVPN-specific Extended Communities.

   The Gateway PE SHOULD NOT copy the above Extended Community types from the original ISF route into the re-advertised ISF route.

5. For a given ISF route, only the BGP Path Attributes associated with the best path MAY be propagated when re-advertising the route into a different domain. If multiple paths are received for the same prefix within the same ISF SAFI, the standard BGP best path selection procedure MUST be applied to determine the active path and its associated attributes. Even when Equal-Cost Multi-Path (ECMP) is enabled for the IP-VRF, only the Path Attributes of the selected best path SHALL be propagated.
"

GV> In above item 1 & 4 is mentuoned that certain attributes and communities SHOULD NOT be propagated. What would break if it is propagated? Would using a stronger normative MUST NOT be more appropriate because with a SHOULD implementor has free choice and interop issue may exist

838	   *  The Community, Extended Community and Large Community attributes
839	      of the aggregate ISF route SHOULD contain all the Communities/
840	      Extended Communities/Large Communities from all of the aggregated
841	      ISF routes, with the exceptions of the extended communities listed
842	      in Section 5.2 that are not propagated.

GV> Is there a reason why wide communities are not mentioned in this list, while they were mentioned at prior section 5.2? 

"
* The Community, Extended Community and Large Community attributes of an aggregated ISF route SHOULD include the union of the corresponding attributes from all constituent ISF routes that were aggregated, with the exception of those Extended Community types explicitly excluded from propagation as specified in Section 5.2.
"

846	   Assuming the aggregation can be performed (the above rules are
847	   applied), the operator should consider aggregation to deal with
848	   scaled tenant networks where a significant number of host routes
849	   exists.  For example, large Data Centers.

GV>

"
If the conditions for route aggregation, as specified above, are satisfied, operators SHOULD consider enabling aggregation in environments with large-scale tenant networks where a significant number of host routes are present. This practice is particularly applicable to deployments such as large-scale data centers.
"

851	6.  Route Selection Process for ISF Routes
852
853	   A PE may receive an IP prefix in ISF routes with different ISF SAFIs,
854	   from the same or different BGP peer.  It may also receive the same IP
855	   prefix (host route) in an EVPN MAC/IP Advertisement route and EVPN IP
856	   Prefix route.  A route selection algorithm across all ISF SAFIs is
857	   needed so that:
858
859	   *  Different gateway and composite PEs have a consistent and
860	      deterministic view on how to reach a given prefix.
861
862	   *  Prefixes advertised in EVPN and other ISF SAFIs can be compared
863	      based on path attributes commonly used by operators across
864	      networks.
865
866	   *  Equal Cost Multi-Path (ECMP) is allowed across EVPN and other ISF
867	      SAFI routes.
868
869	   For a given prefix advertised in one or more non-EVPN ISF routes, the
870	   BGP best path selection procedure will produce a set of "non-EVPN
871	   best paths".  For a given prefix advertised in one or more EVPN ISF
872	   routes, the BGP best path selection procedure will produce a set of
873	   "EVPN best paths".  To support EVPN/non-EVPN ISF interworking in the
874	   context of the same IP-VRF receiving non-EVPN and EVPN ISF routes for
875	   the same prefix, it is then necessary to run a tie-breaking selection
876	   algorithm on the union of these two sets.  This tie-breaking
877	   algorithm begins by considering all EVPN and other ISF SAFI routes,
878	   equally preferable routes to the same destination, and then selects
879	   routes to be removed from consideration.  The process terminates as
880	   soon as only one route remains in consideration.
881
882	   The route selection algorithm must remove from consideration the
883	   routes following the rules and the order defined in [RFC4271], with
884	   the following exceptions and in the following order:
885
886	   1.  Immediately after removing from consideration all routes that are
887	       not tied for having the highest Local Preference, any routes that
888	       do not have the shortest D-PATH are also removed from
889	       consideration.  Routes with no D-PATH are considered to have a
890	       zero-length D-PATH.  A BGP speaker MUST skip this rule for ISF
891	       SAFI routes that are not imported in an IP-VRF.
892
893	   2.  Then regular [RFC4271] selection criteria is followed.
894
895	   3.  At the end of the selection algorithm, if at least one route
896	       still under consideration is an EVPN MAC/IP Advertisement route,
897	       remove from consideration any EVPN IP Prefix routes.
898
899	   4.  If Steps 1-3 leave Equal Cost Multi-Paths (ECMP) between non-EVPN
900	       and EVPN paths, the EVPN path MUST be considered (and the non-
901	       EVPN path removed from consideration).  However, if ECMP across
902	       ISF SAFIs is enabled by policy, and one EVPN path and one non-
903	       EVPN path remain at the end of step 3, both path types MUST be
904	       used.
905
906	   The above process modifies the [RFC4271] selection criteria for
907	   multiprotocol BGP routes with SAFI 128 and EVPN IP Prefix routes to
908	   include the shortest D-PATH so that operators minimize the number of
909	   Gateways and domains through which packets need to be routed.  D-PATH
910	   does not modify the selection process for routes different from SAFI
911	   128 or EVPN routes (received routes with other SAFIs get a treat-as-
912	   withdraw behavior as described in Section 4).
913
914	   Example 1 - PE1 receives the following routes for IP1/32, that are
915	   candidate to be imported into IP-VRF-1:
916
917	   {SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=(100,200)}
918	   {SAFI=EVPN, RT-5, Local-Pref=100, AS-Path=(100,200)}
919	   {SAFI=128, Local-Pref=100, AS-Path=(100,200)}
920
921	   Selected route: {SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=100,200]
922	   (due to step 3, and no ECMP).
956
924	   Example 2 - PE1 receives the following routes for IP2/24, that are
925	   candidate to be imported into IP-VRF-1:
956
927	   {SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS-Path=(100,200), MED=10}
928	   {SAFI=128, D-PATH=(6500:1:EVPN,6500:2:IPVPN), AS-Path=(200), MED=200}
929
930	   Selected route: {SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS-
931	   Path=(100,200), MED=10} (due to step 1).

GV>

"
6. Route Selection Process for ISF Routes

A PE router may receive the same IP prefix via ISF routes with different ISF SAFIs, and from either the same or different BGP peers. Additionally, the same IP prefix (e.g., a host route) may be received in both an EVPN MAC/IP Advertisement route and an EVPN IP Prefix route. To ensure consistent and deterministic forwarding behavior, a route selection procedure across all ISF SAFIs is required.

The objectives of this route selection process are as follows:

* To ensure that all Composite and Gateway PEs have a consistent and deterministic view of the preferred path to reach a given IP prefix.
* To enable meaningful comparison of routes advertised in EVPN and non-EVPN ISF SAFIs based on commonly used path attributes.
* To support Equal-Cost Multi-Path (ECMP) forwarding across EVPN and non-EVPN ISF SAFI routes, where applicable.

For a given prefix received via one or more non-EVPN ISF routes, the standard BGP best path selection procedure, as defined in [RFC4271], is applied to determine the "non-EVPN best paths." Similarly, for a given prefix received via one or more EVPN ISF routes, the same procedure is applied to determine the "EVPN best paths."

When both EVPN and non-EVPN ISF routes are present for the same prefix within a single IP-VRF, the PE MUST perform a tie-breaking selection procedure on the union of these best-path sets. The process treats all candidate ISF routes as equally preferable initially, then iteratively removes routes until a single best path (or a valid ECMP set) remains.

6.1. Tie-Breaking and Selection Rules

The selection procedure MUST follow the standard route selection rules defined in [RFC4271], with the following additional rules and exceptions applied in the specified order:

1. Immediately after applying the Local Preference comparison step from [RFC4271], the PE MUST remove from consideration any routes that do not have the shortest D-PATH attribute. Routes with no D-PATH attribute are considered to have a D-PATH length of zero. This rule MUST NOT be applied to ISF routes that are not imported into an IP-VRF.

2. After applying Rule 1, the standard [RFC4271] selection steps MUST continue in order.

3. If, after the previous steps, one or more candidate routes remain and at least one of them is an EVPN MAC/IP Advertisement route (EVPN Route Type 2), then all EVPN IP Prefix routes (EVPN Route Type 5) MUST be removed from consideration.

4. If ECMP is enabled by policy and the remaining candidate routes after Steps 1 through 3 include both EVPN and non-EVPN paths, then both paths MUST be retained. If ECMP is not enabled, and such a case arises, the EVPN path MUST be selected and the non-EVPN path MUST be removed from consideration.

This procedure extends the standard BGP best path selection behavior as specified in [RFC4271] for SAFI 128 and EVPN IP Prefix routes by incorporating D-PATH based tie-breaking to prefer routes that traverse the fewest Gateway PEs or domains. These rules MUST NOT be applied to routes received under AFI/SAFI combinations other than SAFI 128 or EVPN; such routes are subject to the behavior defined in Section 4, including the "treat-as-withdraw" procedure where applicable.

6.2. Examples

Example 1:
PE1 receives three candidate routes for prefix IP1/32, all eligible for import into IP-VRF-1:

{SAFI=EVPN, RT-2, Local-Pref=100, AS_PATH=(100,200)}
{SAFI=EVPN, RT-5, Local-Pref=100, AS_PATH=(100,200)}
{SAFI=128,        Local-Pref=100, AS_PATH=(100,200)}

Selected route:
{SAFI=EVPN, RT-2, Local-Pref=100, AS_PATH=(100,200)}
This outcome is due to Step 3, which gives preference to Route Type 2 when both Type 2 and Type 5 EVPN routes exist.

Example 2:
PE1 receives two candidate routes for prefix IP2/24, both eligible for import into IP-VRF-1:

{SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS_PATH=(100,200), MED=10}
{SAFI=128,        D-PATH=(6500:1:EVPN, 6500:2:IPVPN), AS_PATH=(200), MED=200}

Selected route:
{SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS_PATH=(100,200), MED=10}
This result is due to Step 1, which prefers the route with the shortest D-PATH.
"

972	   In a composite domain with composite and regular PEs:
973
974	   1.  The composite PEs MUST advertise the same IP prefixes in each ISF
975	       SAFI to the Route Reflector (RR).  For example, in Figure 7, the
976	       prefix IP1/24 is advertised by PE1 and PE2 to the Route Reflector
977	       in two separate NLRIs, one for AFI/SAFI 1/128 and another one for
978	       EVPN.  If the two routes are advertised with the same set of
979	       attributes, the remote Composite PE selection process will pick
980	       up the EVPN route over the IPVPN route (as per Section 6).  For
981	       this reason, prioritizing the announcement of the EVPN route
982	       prior to the IPVPN route is an OPTIONAL optimization, since, if
983	       the EVPN route arrives at the composite PE first, the selected
984	       route is not replaced by the IPVPN route later.
985
986	   2.  As an informative note, the Route Reflector does not forward EVPN
987	       routes to neighbors on which the EVPN SAFI is not enabled, and
988	       similarly, the Route Reflector does not forward IPVPN routes to
989	       neighbors on which the IPVPN SAFI is not enabled.  For example,
990	       the Route Reflector does not forward EVPN routes to PE3 (since
991	       the Route Reflector does not have the EVPN SAFI enabled on its
992	       BGP session to PE3), whereas the IPVPN routes are forwarded to
993	       all the PEs.
994
995	   3.  IPVPN PEs process and import IPVPN routes, as in [RFC4364].  As
996	       an example, PE3 receives only the IPVPN route for IP1/24 and
997	       resolves the BGP next-hop to an MPLS tunnel (with IP payload) to
998	       PE1 and/or PE2.
999
1000	   4.  Composite PEs MUST process routes for the same prefix coming from
1001	       different ISF SAFI routes, and perform route selection.
1002
1003	       *  As an example, PE4 receives IP1/24 encoded in EVPN and another
1004	          ISF SAFI route (EVPN IP Prefix route and IPVPN).  The route
1005	          selection follows the procedures in Section 6.
1006
1007	       *  Assuming an EVPN route is selected, PE4 resolves the BGP next-
1008	          hop to an MPLS tunnel (with Ethernet or IP payload) to PE1
1009	          and/or PE2.  As described in Section 3, two EVPN PEs may use
1010	          tunnels with Ethernet or IP payloads to connect their IP-VRFs,
1011	          depending on the [RFC9136] model implemented.
1012
1013	       *  The other composite PEs (PE1 and PE2) receive also the same IP
1014	          prefix via EVPN and IPVPN SAFIs and they also follow the route
1015	          selection in Section 6.
1016
1017	   5.  When a given route has been selected as the route for a
1018	       particular packet, the transmission of the packet MUST be done
1019	       according to the rules for that route's AFI/SAFI.
1020
1021	   6.  As an informative note, in composite domains, such as the one in
1022	       Figure 7, the EVPN advanced forwarding features will only be
1023	       available to composite and EVPN PEs (assuming they select an EVPN
1024	       IP Prefix route to forward packets for a given IP prefix), and
1025	       not to IPVPN PEs.  For example, assuming PE1 sends IP1/24 in an
1026	       EVPN and an IPVPN route and the EVPN route is the best one in the
1027	       selection, the recursive resolution of the EVPN IP Prefix routes
1028	       can only be used in PE2 and PE4 (composite PEs), and not in PE3
1029	       (IPVPN PE).  As a consequence of this, the indirection provided
1030	       by the EVPN IP Prefix route recursive resolution and its benefits
1031	       in a scaled network, will not be available in all the PEs in the
1032	       network.

GV> 

"
In a composite domain comprising both composite and regular PE devices, the following behaviors apply:

1. Prefix Advertisement Consistency
   Composite PEs MUST advertise the same IP prefixes using each ISF SAFI to the Route Reflector (RR). For example, as shown in Figure 7, the prefix IP1/24 is advertised by PE1 and PE2 to the Route Reflector in two separate NLRI entries: one for AFI/SAFI 1/128 (IPVPN) and another for EVPN.
   If both routes are advertised with the same set of BGP Path Attributes, the receiving composite PE will select the EVPN route over the IPVPN route, following the route selection procedures defined in Section 6.
   While not required, prioritizing the advertisement of the EVPN route before the IPVPN route is an OPTIONAL optimization. This ensures that the EVPN route is more likely to be selected first, avoiding unnecessary replacement if the IPVPN route arrives later.

2. Route Reflector SAFI-Specific Forwarding Behavior
   The Route Reflector MUST NOT forward EVPN routes to peers for which the EVPN SAFI is not enabled, and likewise MUST NOT forward IPVPN routes to peers lacking IPVPN SAFI support.
   For instance, in Figure 7, the Route Reflector does not forward EVPN routes to PE3 if the EVPN SAFI is not enabled on its BGP session with PE3. However, the IPVPN routes are forwarded to all PEs that have IPVPN SAFI enabled.

3. IPVPN PE Route Processing
   Regular IPVPN PEs MUST process and import IPVPN routes as specified in [RFC4364].
   For example, PE3 receives only the IPVPN route for prefix IP1/24 and resolves the BGP next-hop to establish an MPLS tunnel with IP payload toward PE1 and/or PE2.

4. Composite PE Route Selection
   Composite PEs MUST perform route selection for prefixes received via multiple ISF SAFIs, applying the procedures described in Section 6.

   * For example, PE4 receives prefix IP1/24 via both an EVPN route and a non-EVPN ISF route (e.g., an IPVPN route). Route selection is performed as specified in Section 6.

   * If the EVPN route is selected, PE4 resolves the BGP next-hop to an MPLS tunnel, which may carry either Ethernet or IP payloads to PE1 and/or PE2. As described in Section 3, the tunnel type used between EVPN PEs depends on the encapsulation model supported, such as that defined in [RFC9136].

   * Other composite PEs (e.g., PE1 and PE2) receiving the same prefix via both EVPN and IPVPN SAFIs must also apply the route selection process defined in Section 6.

5. Forwarding Behavior Based on Selected Route
   Once a route has been selected for a given IP prefix, packet forwarding MUST follow the forwarding rules associated with the AFI/SAFI of the selected route.

6. Applicability of EVPN Forwarding Enhancements
   In composite domains such as the one depicted in Figure 7, the advanced forwarding features provided by EVPN are available only to composite and EVPN-capable PEs that select an EVPN IP Prefix route as the best path. These enhancements are not available to IPVPN-only PEs.
   For example, if PE1 advertises IP1/24 using both EVPN and IPVPN routes, and the EVPN route is selected as the best path, only composite PEs such as PE2 and PE4 can leverage EVPN-specific recursive resolution and forwarding mechanisms. IPVPN PEs, such as PE3, cannot utilize these capabilities.
   Consequently, the benefits of EVPN-based indirection and route resolution in large-scale deployments may not be available uniformly across all PEs in the network.
"

1034	8.  Gateway PE Procedures
1035
1036	   Section 3 defines a gateway PE as an Interworking PE that is attached
1037	   to two (or more) domains and propagates ISF routes between those
1038	   domains.  Examples of gateway PEs are Data Center gateways connecting
1039	   domains that make use of EVPN and other ISF SAFIs for a given tenant.
1040	   The gateway PE procedures in this document provide an interconnect
1041	   solution for ISF routes and complement the gateway definition of
1042	   [RFC9014], which focuses on the interconnect solution for Layer 2.
1043	   This section applies to the interconnect of two domains that use
1044	   different ISF SAFIs (e.g., EVPN to IPVPN), as well as the
1045	   interconnect of two domains of the same ISF SAFI (e.g., EVPN to
1046	   EVPN).  Figure 8 illustrates a gateway PE use-case, in which PE1 and
1047	   PE2 (and PE3/PE4) are gateway PEs interconnecting domains for the
1048	   same tenant.

GV>

"
8. Gateway PE Procedures

As defined in Section 3, a Gateway PE is an Interworking PE that connects two or more domains and facilitates the propagation of ISF routes between those domains. Typical examples include data center gateway devices that interconnect domains utilizing different ISF SAFIs, such as EVPN and IPVPN, for the same tenant network.

The gateway PE procedures specified in this document define the mechanisms required to support ISF route interconnection across such domains. These procedures extend the concept of a gateway PE beyond the scope of [RFC9014], which focuses on Layer 2 interconnection, by providing an analogous interconnection model for ISF route exchange at Layer 3.

The procedures described in this section apply to both of the following scenarios:

* Interconnection between domains utilizing different ISF SAFIs (e.g., EVPN to IPVPN).
* Interconnection between domains utilizing the same ISF SAFI (e.g., EVPN to EVPN).

Figure 8 provides an illustrative example of this model, wherein PE1 and PE2 (as well as PE3 and PE4) operate as Gateway PEs interconnecting different domains associated with the same tenant.
"

1075	   The procedures for a gateway PE enabled for ISF SAFI-x and ISF SAFI-y
1076	   on the same IP-VRF follow:
1077
1078	   1.  A gateway PE that imports an ISF SAFI-x route to prefix P in an
1079	       IP-VRF, MUST export P in ISF SAFI-y if:
1080
1081	       a.  P is installed in the IP-VRF - which means the SAFI-x route
1082	           is well-formed, valid and the best one for P - and
1083
1084	       b.  PE has a BGP peer for SAFI-y (enabled for the same IP-VRF)
1085	           and
1086
1087	       c.  The advertisement is allowed by policy and
1088
1089	       d.  ISF SAFI-x and ISF SAFI-y are any of the types defined in
1090	           Section 3.  Note that SAFI-x and SAFI-y MAY have the same
1091	           value.
1092
1093	       In the example of Figure 8, gateway PE1 and PE2 receive an EVPN
1094	       IP Prefix route with IP1/24, install the prefix in the IP-VRF and
1095	       re-advertise it using SAFI 128.
1096
1097	   2.  A gateway PE that receives an ISF SAFI-x route to prefix P in an
1098	       IP-VRF MUST NOT export P in ISF SAFI-y if:
1099
1100	       a.  The ISF SAFI-x route is not well-formed or valid.  Rules to
1101	           determine if a route is well-formed or valid for a given ISF
1102	           SAFI are defined by the specification of each ISF SAFI.  As
1103	           an example, an EVPN IP Prefix route received with non-zero
1104	           ESI and GW IP values, at the same time, is not valid as per
1105	           [RFC9136], section 3.2.
1106
1107	       b.  The ISF SAFI-x route contains a D-PATH attribute with one or
1108	           more of the gateway PE's locally associated domains for the
1109	           IP-VRF.  In this case the route is considered to be a looped
1110	           ISF route, as described in Section 4 and hence MUST NOT be
1111	           exported in ISF SAFI-y.
1112
1113	   Once the gateway PE determines that P must be exported, P will be
1114	   advertised using ISF SAFI-y as follows:
1115
1116	   a.  If Uniform-Propagation-Mode is enabled Section 5.2, the D-PATH
1117	       attribute MUST be included if SAFI-y is equal to 128 or EVPN, so
1118	       that loops can be detected in remote gateway PEs.  When a gateway
1119	       PE propagates an ISF route between domains, it MUST prepend a
1120	       <DOMAIN-ID:ISF_SAFI_TYPE> to the received D-PATH attribute.  The
1121	       DOMAIN-ID and ISF_SAFI_TYPE fields refer to the domain over which
1122	       the gateway PE received the IP prefix and the ISF SAFI of the
1123	       route, respectively.  If the received IP prefix route did not
1124	       include any D-PATH attribute, the gateway IP MUST add the D-PATH
1125	       when readvertising.  The D-PATH in this case will have only one
1126	       segment on the list, the <DOMAIN-ID:ISF_SAFI_TYPE> of the
1127	       received route.
1128
1129	       In the example of Figure 8, gateway PE1/PE2 receive the EVPN IP
1130	       Prefix route with no D-PATH attribute since the route is
1131	       originated at PE5.  Therefore PE1 and PE2 will add the D-PATH
1132	       attribute including <DOMAIN-ID:ISF_SAFI_TYPE> = <6500:1:EVPN>.
1133	       Gateways PE3/PE4 will propagate the route again, now prepending
1134	       their <DOMAIN-ID:ISF_SAFI_TYPE> = <6500:2:IPVPN>.  PE6 receives
1135	       the EVPN IP Prefix routes with D-PATH =
1136	       {<6500:2:IPVPN>,<6500:1:EVPN>} and can use that information to
1137	       make BGP path decisions.
1138
1139	   b.  The gateway PE MAY use the Route Distinguisher of the IP-VRF to
1140	       readvertise P in the ISF SAFI-y.
1141
1142	   c.  The label allocation used by each gateway PE is a local
1143	       implementation matter.  The IP-VRF advertising IP prefixes for
1144	       EVPN and another ISF SAFI may use a label per-VRF, per-prefix,
1145	       etc.
1146
1147	   d.  The gateway PE MUST be able to use the same or different set of
1148	       Route Targets per domain on the same IP-VRF.  In particular, if
1149	       different domains use different set of Route Targets for the same
1150	       tenant, the gateway PE MUST be able to import and export routes
1151	       with the different sets.
1152
1153	   e.  Even though Figure 8 only shows two domains per gateway PE, the
1154	       gateway PEs may be connected to more than two domains.
1155
1156	   f.  There is no limitation of gateway PEs that a given IP prefix P
1157	       can pass through until it reaches a given PE.
1158
1159	   g.  If the gateway PE uses Uniform-Propagation-Mode for BGP Path
1160	       Attribute propagation, besides the processing of D-PATH described
1161	       in point "a", the rules in Section 5.2 are followed.
1162
1163	   h.  As an informative note, if P was originated in an EVPN domain but
1164	       traversed a different ISF SAFI domain (or domains), it will lose
1165	       EVPN-specific attributes that are used in advanced EVPN
1166	       procedures.  For example, even if PE1 advertises IP1/24 along
1167	       with a given non-zero ESI (for recursive resolution to that ESI),
1168	       when PE6 receives the IP prefix in an EVPN route, the ESI value
1169	       will be zero.  This is because the route traverses an ISF SAFI
1170	       domain that is different from EVPN.

GV>

"
A Gateway PE that is enabled for two ISF SAFIs, referred to here as SAFI x and SAFI y, on the same IP-VRF, MUST follow the procedures described below for propagating routes between domains.

8.1. Export Conditions

1. A Gateway PE that imports an ISF SAFI x route for prefix P into an IP-VRF MUST export P using ISF SAFI y if all of the following conditions are met:

   a. The route for P is installed in the IP-VRF, indicating that the SAFI x route is well-formed, valid, and selected as the best route.

   b. The PE has an active BGP session with a peer supporting SAFI y, enabled for the same IP-VRF.

   c. Export policy permits the advertisement of the route.

   d. SAFI x and SAFI y are valid ISF SAFIs as defined in Section 3. SAFI x and SAFI y MAY be the same.

   Example: In Figure 8, Gateway PEs PE1 and PE2 receive an EVPN IP Prefix route for prefix IP1/24, install the route in their respective IP-VRFs, and re-advertise it using SAFI 128 (IPVPN).

2. A Gateway PE that receives an ISF SAFI x route for prefix P into an IP-VRF MUST NOT export P using SAFI y under any of the following conditions:

   a. The SAFI x route is not well-formed or valid. Criteria for route validity are defined in the corresponding ISF SAFI specification. For example, an EVPN IP Prefix route that contains both a non-zero EESI and a Gateway IP address is invalid, as specified in [RFC9136], Section 3.2.

   b. The D-PATH attribute of the SAFI x route includes one or more DOMAIN-ID values locally configured on the Gateway PE for the associated IP-VRF. In this case, the route is considered a looped ISF route, as described in Section 4, and MUST NOT be exported using SAFI y.

8.2. Advertisement Behavior

If the export conditions are satisfied, the Gateway PE MUST advertise prefix P using ISF SAFI y in accordance with the following procedures:

a. If Uniform Propagation Mode (see Section 5.2) is enabled, the Gateway PE MUST include the D-PATH attribute when SAFI y is either SAFI 128 (IPVPN) or EVPN. This enables loop detection at downstream Gateway PEs.

When propagating an ISF route, the Gateway PE MUST prepend a <DOMAIN-ID:ISF_SAFI_TYPE> element to the received D-PATH attribute. The DOMAIN-ID reflects the domain from which the route was received, and the ISF_SAFI_TYPE reflects the SAFI of the received route.

If the received route does not include a D-PATH attribute, the Gateway PE MUST create and attach a new D-PATH attribute containing a single segment: the <DOMAIN-ID:ISF_SAFI_TYPE> corresponding to the received route.

Example: In Figure 8, Gateway PEs PE1 and PE2 receive an EVPN IP Prefix route from PE5 that does not include a D-PATH attribute. PE1 and PE2 prepend <6500:1:EVPN> to form the new D-PATH. Gateway PEs PE3 and PE4, upon re-advertising the route, prepend <6500:2:IPVPN>, resulting in PE6 receiving the route with D-PATH {<6500:2:IPVPN>, <6500:1:EVPN>}. This information is then used by PE6 in BGP path selection.

b. The Gateway PE MAY use the Route Distinguisher (RD) of the IP-VRF when re-advertising prefix P via ISF SAFI y.

c. Label allocation for route advertisement is an implementation-specific matter. The Gateway PE MAY use per-VRF, per-prefix, or other label allocation models.

d. The Gateway PE MUST support the use of distinct Route Target (RT) sets per domain on the same IP-VRF. If multiple domains associated with a tenant use different RT sets, the Gateway PE MUST be capable of importing and exporting routes according to each domain's RT configuration.

e. Although Figure 8 illustrates a scenario with only two domains per Gateway PE, Gateway PEs MAY interconnect more than two domains.

f. There is no restriction on the number of Gateway PEs that a given prefix P may traverse before reaching its destination.

g. If Uniform Propagation Mode is used for BGP Path Attribute propagation, the Gateway PE MUST follow the procedures defined in Section 5.2 in addition to the D-PATH specific behavior described in item (a).

h. Informative Note: If prefix P is originated in an EVPN domain and subsequently traverses one or more non-EVPN ISF SAFI domains, it will lose EVPN-specific attributes used for advanced EVPN procedures.
For example, if PE1 advertises prefix IP1/24 along with a non-zero ESI (for recursive resolution to that ESI), the ESI value will be reset to zero by the time the route reaches PE6, as it passed through an ISF SAFI domain that is not EVPN-capable. Consequently, certain EVPN-specific functionalities may not be preserved end-to-end.
"

GV> In the above step h, is there a detailed list of functionalities that may b=not be preserved and the implications of this?

1172	9.  Interworking Use-Cases
1173
1174	   While Interworking PE networks may well be similar to the examples
1175	   described in Section 7 and Section 8, in some cases a combination of
1176	   both functions may be required.  Figure 9 illustrates an example
1177	   where the gateway PEs are also composite PEs, since not only they
1178	   need to propagate ISF routes between domains (from EVPN SAFI to IPVPN
1179	   and/or EVPN SAFIs), but they also need to interwork with IPVPN-only
1180	   PEs in a domain with a mix of composite and IPVPN-only PEs.

GV>

"
9. Interworking Use Cases

While network deployments involving Interworking PEs may align with the scenarios described in Sections 7 and 8, there are cases where a combination of both Gateway PE and Composite PE functionality is required.

Figure 9 illustrates an example in which Gateway PEs also operate as Composite PEs. In such scenarios, the devices must not only propagate ISF routes between domains, such as between EVPN and IPVPN SAFIs or across multiple EVPN domains, but also interoperate with IPVPN-only PEs within domains that include a mix of Composite and IPVPN-only PEs.
"

1212	   In the example above, PE1 and PE2 MUST follow the procedures
1213	   described in Section 7 and Section 8.  Compared to the example in
1214	   Section 8, PE1 and PE2 now need to also propagate ISF routes from
1215	   EVPN to EVPN, in addition to propagating prefixes from EVPN to IPVPN.

1217	   It is worth noting that PE1 and PE2 will receive TS4's IP prefix via
1218	   IPVPN and EVPN IP Prefix routes.  When readvertising to NVE1 and
1219	   NVE2, PE1 and PE2 will consider the D-PATH rules and attributes of
1220	   the selected route for TS4 (Section 6 describes the Route Selection
1221	   Process).

GV>

"
In the example illustrated, PE1 and PE2 MUST follow the procedures defined in Sections 7 and 8. Unlike the scenario described in Section 8, PE1 and PE2 are additionally required to propagate ISF routes between EVPN domains (i.e., EVPN-to-EVPN), in addition to EVPN-to-IPVPN propagation.

It is important to note that PE1 and PE2 will receive the IP prefix associated with TS4 via both IPVPN and EVPN IP Prefix routes. When re-advertising the selected route to NVE1 and NVE2, PE1 and PE2 MUST apply the D-PATH handling rules and related attribute processing as described in Section 6 (Route Selection Process).
"

1223	10.  BGP Error Handling on Interworking PEs
1224
1225	   An Interworking PE (acting as gateway PE or composite PE) observes
1226	   the following error-handling procedures for ISF routes:
1227
1228	   *  An UPDATE message for an ISF route containing a D-PATH attribute
1229	      MUST follow the error-handling rules for D-PATH, as specified in
1230	      Section 4.
1231
1232	   *  Any received UPDATE for an ISF route complies with the procedures
1233	      in [RFC7606].
1234
1235	   *  The Interworking PEs do not introduce any new error-handling rules
1236	      for UPDATES received with NLRIs and BGP Path Attributes defined in
1237	      other specifications.  Implementors should refer to the
1238	      appropriate error handling documents for each of the supported
1239	      route types.  These include:
1240
1241	         BGP IP routes: [RFC4760], [RFC8950].
1242
1243	         IPVPN routes: [RFC4364], [RFC4659].
1244
1245	         EVPN MAC/IP routes (RT-2): [RFC7432], [RFC8365].
1246
1247	         EVPN IP Prefix routes (RT-5): [RFC9136].
1248
1249	         Received UPDATE messages to be programmed in IP-VRFs supporting
1250	         Segment Routing for IPv6 data path (SRv6): [RFC9252].
1251
1252	   If a gateway PE is set to propagate BGP Path Attributes for ISF
1253	   routes across domains, the procedures in Section 5.2 guarantee that a
1254	   BGP speaker does not receive UPDATES with well-formed but unexpected
1255	   BGP Path Attributes.  If a gateway PE fails to follow the propagation
1256	   rules in Section 5.2 and propagates some BGP Path Attributes
1257	   erroneously, the receiving PEs follow the specifications for the
1258	   specific ISF route type and BGP Path Attribute.  Some (but not all)
1259	   examples follow:
1260
1261	   *  If the gateway PE erroneously propagates the Router's MAC Extended
1262	      Community [RFC9135] from an EVPN domain to another EVPN domain,
1263	      the receiving PE may find two EVPN Router's MAC extended
1264	      communities in the same ISF route.  In this case, the PE follows
1265	      [RFC9135] and processes the first one (ignoring the second
1266	      extended community).
1267
1268	   *  If the gateway PE erroneously propagates the BGP Encapsulation
1269	      Extended Community (or equivalent Encapsulation TLV in the Tunnel
1270	      Encapsulation Attribute) [RFC9012] from an EVPN domain to another
1271	      EVPN domain, the receiving PE may find two BGP Encapsulation
1272	      Extended Communities with different values in the same ISF route.
1273	      The PE in this case follows [RFC8365], which allows multiple
1274	      encapsulations being signaled in the route.  As per [RFC9012],
1275	      encapsulations advertised using the Tunnel Encapsulation attribute
1276	      are considered equally with those advertised using the
1277	      Encapsulation Extended Community.
1278
1279	   *  If the gateway PE erroneously propagates any EVPN extended
1280	      community from an EVPN domain into an IPVPN domain, the receiving
1281	      IPVPN PE ignores the EVPN extended communities, since their
1282	      semantics do not apply to the IPVPN SAFI.
1283
1284	   *  If the gateway PE erroneously propagates a BGP Prefix-SID
1285	      attribute with SRv6 Service TLVs [RFC9252] for an ISF route
1286	      propagated between domains, the receiving PE follows [RFC9252] in
1287	      case multiple SRv6 TLV instances are received.

GV>

"
10. BGP Error Handling on Interworking PEs

An Interworking PE, whether operating in a Gateway PE or Composite PE role, MUST adhere to the following error-handling procedures when processing Inter-Subnet Forwarding (ISF) routes:

* Any BGP UPDATE message for an ISF route that includes a D-PATH Path Attribute MUST be handled in accordance with the error-handling rules defined in Section 4 of this document.

* All received BGP UPDATE messages for ISF routes MUST conform to the general error-handling procedures specified in [RFC7606].

* This specification introduces no new error-handling behaviors for BGP UPDATE messages that contain NLRI and BGP Path Attributes defined in other specifications. Implementations SHOULD apply the relevant error-handling rules specified for each supported route type. Applicable references include:

  * BGP IP routes: [RFC4760], [RFC8950]
  * IPVPN routes: [RFC4364], [RFC4659]
  * EVPN MAC/IP Advertisement routes (Route Type 2): [RFC7432], [RFC8365]
  * EVPN IP Prefix routes (Route Type 5): [RFC9136]
  * ISF routes installed in IP-VRFs with SRv6 forwarding: [RFC9252]

If a Gateway PE is configured to propagate BGP Path Attributes for ISF routes between domains, the procedures specified in Section 5.2 are intended to ensure that receiving BGP speakers do not encounter UPDATE messages containing well-formed but semantically inappropriate BGP Path Attributes. However, if a Gateway PE incorrectly propagates such attributes in violation of the procedures in Section 5.2, receiving PEs MUST apply the error-handling rules defined in the applicable specifications for the relevant route type and attribute.

The following are examples of such scenarios and their handling:

* If a Gateway PE erroneously propagates the Router's MAC Extended Community [RFC9135] from one EVPN domain to another, the receiving PE MAY observe two instances of this attribute in the same ISF route. In this case, the receiving PE MUST follow the behavior defined in [RFC9135] and MUST process only the first instance, ignoring the second.

* If a Gateway PE erroneously propagates the BGP Encapsulation Extended Community or the equivalent Encapsulation TLV in the Tunnel Encapsulation Attribute [RFC9012] from one EVPN domain to another, the receiving PE MAY receive two encapsulation indications with different values. In such a case, the PE MUST follow the procedures in [RFC8365], which permit signaling multiple encapsulation types. As specified in [RFC9012], encapsulations carried via the Tunnel Encapsulation Attribute MUST be treated as equivalent to those conveyed via the Encapsulation Extended Community.

* If a Gateway PE propagates an EVPN Extended Community from an EVPN domain into an IPVPN domain, the receiving IPVPN PE MUST ignore such communities, as their semantics are not applicable to the IPVPN SAFI.

* If a Gateway PE erroneously propagates a BGP Prefix-SID attribute containing SRv6 Service TLVs [RFC9252] for an ISF route between domains, and the receiving PE receives multiple SRv6 TLV instances, it MUST apply the procedures specified in [RFC9252] for resolving multiple TLVs.
"

1289	11.  Conclusion
1290
1291	   This document describes the procedures required in PEs that process
1292	   and advertise ISF routes for a given tenant.  In particular, this
1293	   document defines:
1294
1295	   *  A route selection algorithm so that a PE can determine what path
1296	      to choose between EVPN paths and other ISF SAFI paths.
1297
1298	   *  A new BGP Path attribute called D-PATH that provides loop
1299	      protection and visibility on the domains a particular route has
1300	      traversed.
1301
1302	   *  The way BGP Path Attributes should be propagated between domains.
1303
1304	   *  The procedures that must be followed on Interworking PEs that
1305	      behave as composite PEs, gateway PEs or a combination of both.
1306
1307	   The above procedures provide an operator with the required tools to
1308	   build large tenant networks that may span multiple domains, use
1309	   different ISF SAFIs to handle IP prefixes, in a deterministic way and
1310	   with routing loop protection.

GV>

"
11. Conclusion

This document defines procedures applicable to PE routers that process and advertise ISF routes for a given tenant. In particular, the following key procedures are specified:

* A route selection algorithm that enables a PE to deterministically select a best path among candidates learned via EVPN and other ISF SAFIs.

* A new BGP Path Attribute, referred to as the Domain Path (D-PATH) attribute, which provides loop prevention capabilities and conveys domain traversal information for a given route.

* The rules governing BGP Path Attribute propagation across domains to maintain semantic consistency and enable cross-domain route processing.

* The operational procedures required on Interworking PEs that function as Composite PEs, Gateway PEs, or devices supporting both roles.

Collectively, these procedures equip operators with the necessary mechanisms to deploy scalable tenant networks spanning multiple administrative or routing domains, employing different ISF SAFIs for IP prefix dissemination while maintaining deterministic forwarding behavior and routing loop protection.
"

1312	12.  Security Considerations
1313
1314	   In general, the security considerations described in [RFC9136] and
1315	   [RFC4364] apply to this document.
1316
1317	   Section 4 introduces the use of the D-PATH attribute, which provides
1318	   a loop prevention mechanism that is used by gateway PEs that
1319	   propagate ISF IPVPN/EVPN routes between domains.  A correct use of
1320	   the D-PATH will prevent control plane and data plane loops in the
1321	   network, however an incorrect configuration of the DOMAIN-IDs or an
1322	   inconsistent support of D-PATH on the Gateway PEs may lead to the
1323	   detection of false route loops, the blackholing of the traffic or may
1324	   result in inconsistent and sub-optimal routing.  An attacker may
1325	   benefit of this transitive attribute to propagate the wrong domain
1326	   information across multiple domains.
1327
1328	   Section 4 restricts the use of D-PATH to IPVPN and EVPN routes in
1329	   "walled garden" Virtual Private Networks.  An upgraded PE removes
1330	   D-PATH from the BGP Path Attributes before advertising an IP Prefix
1331	   to a CE in a SAFI 1 route.  However, if D-PATH is received by a non-
1332	   upgraded IPVPN PE that has an attached CE connected to the Internet,
1333	   the PE may incorrectly propagate the D-PATH attribute in a SAFI 1
1334	   route to the CE, and the D-PATH attribute may then escape out of the
1335	   "walled garden" to the Internet.  This may happen when the IPVPN PE
1336	   re-exports a route directly, or via route leaking between IP-VRFs.
1337	   Since D-PATH is a transitive attribute, if not upgraded to understand
1338	   D-PATH, the CE may propagate the attribute to the Internet.  However,
1339	   since the attribute does not change the best path selection for SAFI
1340	   1 routes, D-PATH cannot create loops or inconsistent routing in the
1341	   Internet.  Upgraded Internet routers receiving the D-PATH attribute
1342	   in a SAFI 1 route will apply the treat-as-withdraw behavior, as
1343	   discussed in Section 4.  As an additional security mechanism, a PE
1344	   following this specification that receives an ISF EVPN or IPVPN route
1345	   from a non-upgraded PE should discard the route via policy if the
1346	   route contains the D-PATH attribute.
1347
1348	   In addition, Section 5.2 introduces the propagation of BGP Path
1349	   Attributes between domains on gateway PEs.  Without this mode of
1350	   propagation, BGP Path Attributes are re-initialized when re-exporting
1351	   ISF routes into a different domain, and the operator does not have
1352	   the end-to-end visibility of a given ISF route path.  However, the
1353	   Uniform Propagation mode introduces the capability of propagating BGP
1354	   Path Attributes beyond the ISF SAFI scope.  While this is a useful
1355	   tool to provide end-to-end visibility across multiple domains, it can
1356	   also be used by an attacker to propagate wrong (although correctly
1357	   formed) BGP Path Attributes that can influence the BGP path selection
1358	   in remote domains.  An implementation can also choose Section 5.1
1359	   (No-propagation mode) to minimize the risks derived from propagating
1360	   incorrect attributes, but again, this mode of operation will prevent
1361	   the receiver PE from seeing the attributes that the originator of the
1362	   route intended to convey in the first place.

GV>

"
12. Security Considerations

The security considerations outlined in [RFC9136] and [RFC4364] are applicable to this specification.

This document introduces the D-PATH Path Attribute (Section 4), which provides a mechanism for control-plane loop prevention when ISF IPVPN and EVPN routes are propagated across multiple domains via Gateway PEs. When configured and supported correctly, the use of the D-PATH attribute helps prevent both control-plane and data-plane loops. However, incorrect configuration of DOMAIN-ID values or inconsistent support for D-PATH among Gateway PEs may result in false-positive loop detection, traffic blackholing, or suboptimal and inconsistent routing behavior. Furthermore, as D-PATH is a transitive BGP attribute, a malicious actor may attempt to inject incorrect domain information that propagates across multiple administrative boundaries.

To mitigate such risks, the use of D-PATH is explicitly restricted to IPVPN and EVPN routes within "walled garden" Virtual Private Networks, as specified in Section 4. A PE that conforms to this specification MUST remove the D-PATH attribute prior to advertising a prefix to a CE router in a SAFI 1 (NLRI used for unicast forwarding) route. If a non-upgraded PE that does not support D-PATH receives such a route and is connected to a CE with Internet access, it may erroneously propagate the D-PATH attribute in a SAFI 1 UPDATE to the CE. If the CE further propagates the route, the D-PATH attribute could inadvertently escape into the public Internet.

However, the presence of the D-PATH attribute in SAFI 1 routes MUST NOT impact BGP best-path selection for those routes and, as such, cannot introduce routing loops or instability in the Internet. Additionally, BGP speakers beyond the "walled garden" that support D-PATH and receive the attribute in SAFI 1 routes MUST apply the "treat-as-withdraw" behavior, as described in Section 4 and consistent with [RFC7606].

As a further safeguard, implementations SHOULD enforce local policy on upgraded PEs to discard any ISF EVPN or IPVPN routes received from non-upgraded peers if such routes include a D-PATH attribute, to prevent unintended propagation.

Section 5.2 of this document introduces Uniform Propagation Mode, which enables Gateway PEs to propagate a consistent set of BGP Path Attributes across domain boundaries. This mode enhances operational visibility by preserving attributes end-to-end along the route path. However, it also introduces the possibility that an attacker could inject malformed or semantically inappropriate, but syntactically correct, attributes that influence BGP path selection in remote domains.

To mitigate this risk, an operator MAY choose to deploy No-Propagation Mode (Section 5.1), wherein BGP Path Attributes are re-initialized upon domain transition. While this limits attribute-based attack vectors, it also eliminates the ability of downstream PEs to inspect the original set of BGP Path Attributes as intended by the route originator.

Operators SHOULD carefully weigh the trade-offs between visibility and control when selecting the appropriate propagation mode and ensure that policies are in place to validate attribute contents at domain boundaries.
"

Kind Regards,
Gunter Van de Velde
Routing Area Director