[bess] [Shepherding AD review] review of draft-ietf-bess-evpn-irb-extended-mobility-17

["Gunter van de Velde (Nokia)" <[e-mail]>]

# Gunter Van de Velde, RTG AD, comments for draft-ietf-bess-evpn-irb-extended-mobility-17

Many thanks to Donald Eastlake for his Early RTGDIR review for draft-ietf-bess-evpn-irb-extended-mobility-10. The review improved the document quality. The enhancements to the document have been resolved to satisfaction according https://mailarchive.ietf.org/arch/msg/bess/ZAqF_aVRjbYi9H2hs6SgeZVU2-8/

Thanks to Stephane Litkowski for his detailed Shepherd write-up.

The document technical content seems complete, although it was not always easy to process. I tried to proposed alternate textblobs to improve readability while trying to keep the original message content. During this review process i noted observations when something was not clear and needed potentially clarification or correction.

Great success to learn that there are at least 2 implementations of the draft

#issues
#======
# This document is not mentioning SRv6 anywhere. Is that the intent? If yes, then maybe that should be explicit mentioned early in the document. 

# figure 3 seems to be missing some components: PE3, PE4 and ESI-2?

# The abstract seems overly detailed. Please consider the suggested alternate abstract, higher level and more generic, with focus upon detailing more abstract the objective of draft-ietf-bess-evpn-irb-extended-mobility-17


#DETAILED COMMENTS
#=================
##classified as [minor] and [major] and [re-edit]

17	Abstract

19	   The procedure to handle host mobility in a layer 2 Network with EVPN
20	   control plane is defined as part of RFC7432.  EVPN has since evolved
21	   to find wider applicability across various IRB use cases that include
22	   distributing both MAC and IP reachability via a common EVPN control
23	   plane.  MAC Mobility procedures defined in RFC7432 are extensible to
24	   IRB use cases if a fixed 1:1 mapping between host IP and MAC is
25	   assumed across host moves.  Generic mobility support for IP and MAC
26	   addresses that allows these bindings to change across moves IS
27	   REQUIRED to support a broader set of EVPN IRB use cases.  EVPN all-
28	   active multi-homing further introduces scenarios that require
29	   additional consideration from mobility perspective.  This document
30	   enumerates a set of design considerations applicable to mobility
31	   across these EVPN IRB use cases and updates sequence number
32	   assignment procedures defined in RFC7432 to address these IRB use
33	   cases.

[major]
This abstract is very detailed and makes it hard to understand on a high level what exactly the content of the draft is all about. I view upon a abstract as the textblob one gives to your people manager to make him/het understand what the document is all about. What about the following abstract textblob proposal, making high level draft intent better understandable for non-EVPN technology wizards

"
This document specifies extensions to RFC7432 Ethernet VPN (EVPN) Integrated Routing and Bridging (IRB) procedures to enhance the mobility mechanisms for EVPN IRB-based networks. The proposed extensions improve the handling of IP address mobility across EVPN networks by introducing a mechanism to track the movement of IP addresses and ensure seamless forwarding. These enhancements address the limitations in the existing EVPN IRB mobility procedures by providing more efficient and scalable solutions. The extensions are backward compatible with existing EVPN IRB implementations and aim to optimize network performance in scenarios involving frequent IP address mobility.
"

127	1.  Introduction
128
129	   EVPN-IRB enables advertising both MAC and IP routes via a single
130	   MAC+IP RT-2 advertisement.  The MAC address is imported into the
131	   local bridge MAC table and enables L2 bridged traffic across the
132	   network overlay.  The IP address is imported into the local ARP table
133	   in an asymmetric IRB design or imported into the IP routing table in
134	   a symmetric IRB design, and enables routed traffic across the layer 2
135	   network overlay.  Please refer to [RFC9135] for more background on
136	   EVPN IRB forwarding modes.

[re-edit]
EVPN-IRB facilitates the advertisement of both MAC and IP routes via a single MAC+IP Route Type 2 (RT-2) advertisement. The MAC address is integrated into the local bridge MAC table, enabling Layer 2 (L2) bridged traffic across the network overlay. The IP address is incorporated into the local ARP table in an asymmetric IRB design, or into the IP routing table in a symmetric IRB design, facilitating routed traffic across the L2 network overlay. For additional context on EVPN IRB forwarding modes, refer to [RFC9135].

138	   To support EVPN mobility procedure, a single sequence number mobility
139	   attribute is advertised with the combined MAC+IP route.  A single
140	   sequence number advertised with the combined MAC+IP route to resolve
141	   both MAC and IP reachability implicitly assumes a 1:1 fixed mapping
142	   between IP and MAC.  While a fixed 1:1 mapping between IP and MAC is
143	   a common use case that is addressed via existing MAC mobility
144	   procedure defined in [RFC7432], additional IRB scenarios need to be
145	   considered, that don't necessarily adhere to this assumption.  Such
146	   use cases are common in a virtualized host environment where hosts
147	   attached to an EVPN network are virtual machines (VM) or
148	   containerized workloads.  Following IRB mobility scenarios are
149	   considered:
150
151	   *  VM move results in VM IP and MAC moving together
152
153	   *  VM move results in VM IP moving to a new MAC association
154
155	   *  VM move results in VM MAC moving to a new IP association

[re-edit]
To support the EVPN mobility procedure, a single sequence number mobility attribute is advertised with the combined MAC+IP route. This approach, which resolves both MAC and IP reachability with a single sequence number, inherently assumes a fixed 1:1 mapping between IP and MAC. While this fixed 1:1 mapping is a common use case and is addressed via the existing MAC mobility procedure defined in [RFC7432], there are additional IRB scenarios that do not adhere to this assumption. Such scenarios are prevalent in virtualized host environments where hosts connected to an EVPN network are virtual machines (VMs) or containerized workloads. The following IRB mobility scenarios are considered:

* A VM move results in the VM's IP and MAC moving together.

* A VM move results in the VM's IP moving to a new MAC association.

* A VM move results in the VM's MAC moving to a new IP association.

157	   While existing MAC mobility procedure can be used for MAC+IP move in
158	   the first scenario, subsequent scenarios result in a new MAC- IP
159	   association.  As a result, a single sequence number assigned
160	   independently per-{MAC, IP} is not sufficient to determine most
161	   recent reachability for both MAC and IP, unless the sequence number
162	   assignment algorithm allows for changing MAC-IP bindings across
163	   moves.
163
165	   This document updates sequence number assignment procedures defined
166	   in [RFC7432] to adequately address mobility support across EVPN-IRB
167	   overlay use cases that allow MAC-IP bindings to change across VM
168	   moves and can support mobility for both MAC and IP components carried
169	   in an EVPN RT-2 for these use cases.

[re-edit]
While the existing MAC mobility procedure can manage the MAC+IP move in the first scenario, the subsequent scenarios lead to new MAC-IP associations. Therefore, a single sequence number assigned independently per-{MAC, IP} is insufficient to determine the most recent reachability for both MAC and IP unless the sequence number assignment algorithm allows for changing MAC-IP bindings across moves.

This document updates the sequence number assignment procedures defined in [RFC7432] to adequately address mobility support across EVPN-IRB overlay use cases that permit MAC-IP bindings to change across VM moves and support mobility for both MAC and IP components carried in an EVPN RT-2 for these use cases.

171	   In addition, for hosts on an ESI multi-homed to multiple PE devices,
172	   additional procedures are specified to ensure synchronized sequence
173	   number assignments across the multi-homing devices.
174
175	   This document covers mobility for the following cases, independent of
176	   the overlay encapsulation (e.g.: MPLS, NVO Tunnel):
177
178	   *  Symmetric EVPN IRB overlay
179
180	   *  Asymmetric EVPN IRB overlay
181
182	   *  Routed EVPN overlay

[re-edit]
Additionally, for hosts on an ESI multi-homed to multiple PE devices, additional procedures are specified to ensure synchronized sequence number assignments across the multi-homing devices.

This document addresses mobility for the following cases, independent of the overlay encapsulation (e.g., MPLS, NVO Tunnel):

* Symmetric EVPN IRB overlay

* Asymmetric EVPN IRB overlay

* Routed EVPN overlay

184	1.1.  Document Structure
185
186	   Following sections of the document are informative:
187
188	   *  section 3 provides the necessary background and problem statement
189	      being addressed in this document.
190
191	   *  section 4 lists the resulting design considerations for the
192	      document.
193
194	   Following sections of the document are normative:
195
196	   *  section 6 describes the mobility and sequence number assigment
197	      procedures in an EVPN-IRB overlay required to address the
198	      scenarios described in section 4.
199
200	   *  section 7 describes the mobility procedures for a routed overlay
201	      network as opposed to an IRB overlay.
202
203	   *  section 8 describes corresponding duplicate detection procedures
204	      for EVPN-IRB and routed overlays.

[minor]
What about section 5? it exists in the draft. I assume the intent is informational 

217	   *  Underlay: IP or MPLS fabric core network that provides IP or MPLS
218	      routed reachability between EVPN PEs.

[major]
Is SRv6 intentionally missing from this list? 

220	   *  Overlay: VPN or service layer network consisting of EVPN PEs OR
221	      VPN provider-edge (PE) switch-router devices that runs on top of
222	      an underlay routed core.

[major]
I believe that this is ambigious terminology. add RFC references to the base RFC that documents each type of overlay PE

233	   *  Symmetric EVPN-IRB: An overlay fabric first-hop routing
234	      architecture as defined in [RFC9135], wherein, overlay host-to-
235	      host routed inter-subnet flows are routed at both ingress and
236	      egress EVPN PEs.

[re-edit]
Symmetric EVPN-IRB: is a specific design approach used in EVPN-based networks [RFC9135] to handle both Layer 2 (L2) and Layer 3 (L3) forwarding within the same network infrastructure. The key characteristic of symmetric EVPN-IRB is that both ingress and egress PE routers perform routing for inter-subnet traffic.

238	   *  Asymmetric EVPN-IRB: An overlay fabric first-hop routing
239	      architecture as defined in [RFC9135], wherein, overlay host-to-
240	      host routed inter-subnet flows are routed and bridged at ingress
241	      PE and bridged at egress PEs.

[re-edit]
Asymmetric EVPN-IRB: is a design approach used in EVPN-based networks [RFC9135] to handle Layer 2 (L2) and Layer 3 (L3) forwarding. In this approach, only the ingress Provider Edge (PE) router performs routing for inter-subnet traffic, while the egress PE router performs bridging.

248	   *  Ethernet-Segment: physical Ethernet or LAG port that connects an
249	      access device to an EVPN PE, as defined in [RFC7432].

[minor]
s/physical Ethernet/Physical ethernet/

251	   *  EVPN all-active multi-homing: PE-CE all-active multi-homing
252	      achieved via a multi-homed layer-2 LAG interface on a CE with
253	      member links to multiple PEs and related EVPN procedures on the
254	      PEs.

[re-edit]
EVPN all-active multi-homing: is a redundancy and load-sharing mechanism used in EVPN networks. This method allows multiple PE devices to simultaneously provide Layer 2 and Layer 3 connectivity to a single CE device or network segment.

256	   *  RT-2: EVPN route type 2 carrying both MAC and IP reachability.
257
258	   *  RT-5: EVPN route type 5 carrying IP prefix reachability.

[minor]
add references to RFC7432

260	   *  MAC-IP: IP association for a MAC, referred to in this document may
261	      be IPv4, IPv6 or both.

[minor]
Is this specified in a document somewhere, or is this explicit to this document itself?


263	   *  SYNC MAC route: In the context of EVPN multi-homing, this refers
264	      to a local MAC route SYNCed from another PE sharing the same ESI.
265
266	   *  SYNC MAC-IP route: In the context of EVPN multi-homing, this
267	      refers to a local MAC-IP route SYNCed from another PE sharing the
268	      same ESI.
269
270	   *  SYNC MAC sequence number: In the context of EVPN multi-homing,
271	      this refers to sequence number received with a SYNC MAC route.
272
273	   *  SYNC MAC-IP sequence number: In the context of EVPN multi-homing,
274	      this refers to sequence number received with a SYNC MAC-IP route.


[minor]
Is the SYNC something outlined in this draft itself, or is this procedure specified in another document?
I assume this is based upon the priciples of RFC7432 using the MAC Mobility Extended Community

279	3.1.  Optional MAC only RT-2
280
281	   In an EVPN IRB scenario, where a single MAC+IP RT-2 advertisement
282	   carries both IP and MAC routes, a MAC only RT-2 advertisement is
283	   redundant for host MACs that are advertised via MAC+IP RT-2.  As a
284	   result, advertisement of a local MAC only RT-2 is an optional at an
285	   EVPN PE.  This is an important consideration for mobility scenarios
286	   discussed in subsequent sections.  Note that a local MAC and its
287	   assigned sequence number is still maintained locally on a PE, and it
288	   is just the advertisement of this route to other PEs that is
289	   optional.

291	   MAC only RT-2 may still be advertised for non-IP host MACs that are
292	   not advertised via MAC+IP RT-2.

[re-edit]
In an EVPN IRB scenario, where a single MAC+IP RT-2 advertisement carries both IP and MAC routes, a MAC-only RT-2 advertisement becomes redundant for host MACs already advertised via MAC+IP RT-2. Consequently, the advertisement of a local MAC-only RT-2 is optional at an EVPN PE. This consideration is important for mobility scenarios discussed in subsequent sections. It is noteworthy that a local MAC and its assigned sequence number are still maintained locally on a PE, and only the advertisement of this route to other PEs is optional.

MAC-only RT-2 advertisements may still be issued for non-IP host MACs that are not included in MAC+IP RT-2 advertisements.

294	3.2.  Mobility Use Cases
295
296	   This section describes the IRB mobility use cases considered in this
297	   document.  Procedures to address them are covered later in section 6
298	   and section 7.
299
300	   *  Host move results in Host IP and MAC moving together
301
302	   *  Host move results in Host IP moving to a new MAC association
303
304	   *  Host move results in Host MAC moving to a new IP association

[re-edit]
This section outlines the IRB mobility use cases addressed in this document. Detailed procedures to handle these scenarios are provided in Sections 6 and 7.

* A host move results in both the host's IP and MAC addresses moving together.

* A host move results in the host's IP address moving to a new MAC address association.

* A host move results in the host's MAC address moving to a new IP address association.

306	3.2.1.  Host MAC+IP Move
307
308	   This is the baseline case, wherein a host move results in both host
309	   MAC and IP moving together with no change in MAC-IP binding across a
310	   move.  Existing MAC mobility defined in [RFC7432] may be leveraged to
311	   apply to corresponding MAC+IP route to support this mobility
312	   scenario.
313
314	3.2.2.  Host IP Move to new MAC
315
316	   This is the case, where a host move results in VM IP moving to a new
317	   MAC binding.
318
319	3.2.2.1.  VM Reload
320
321	   A host reload or an orchestrated host move that results in a host
322	   being re-spawned at a new location may result in host getting a new
323	   MAC assignment, while maintaining its existing IP address.  This
324	   results in a host IP move to a new MAC binding:
325
326	   IP-a, MAC-a ---> IP-a, MAC-b
327
328	3.2.2.2.  MAC Sharing
329
330	   This takes into account scenarios, where multiple hosts, each with a
331	   unique IP, may share a common MAC binding, and a host move results in
332	   a new MAC binding for the host IP.
333
334	   As an example, hosts running on a single physical server, each with a
335	   unique IP, may share the same physical server MAC.  In yet another
336	   scenario, an L2 access network may be behind a firewall, such that
337	   all hosts IPs on the access network are learnt with a common firewall
338	   MAC.  In all such "shared MAC" use cases, multiple local MAC-IP ARP
339	   entries may be learnt with the same MAC.  A host IP move, in such
340	   scenarios (for example, to a new physical server), could result in
341	   new MAC association for the host IP.
342
343	3.2.2.3.  Problem
344
345	   In both of the above scenarios, a combined MAC+IP EVPN RT-2
346	   advertised with a single sequence number attribute implicitly assumes
347	   a fixed IP to MAC mapping.  A host IP move to a new MAC breaks this
348	   assumption and results in a new MAC+IP route.  If this new MAC+IP
349	   route is independently assigned a new sequence number, the sequence
350	   number can no longer be used to determine most recent host IP
351	   reachability in a symmetric EVPN-IRB design OR the most recent IP to
352	   MAC binding in an asymmetric EVPN-IRB design.
353
354	                        +------------------------+
355	                        | Underlay Network Fabric|
356	                        +------------------------+

358	     +-----+   +-----+      +-----+   +-----+      +-----+   +-----+
359	     | PE1 |   | PE2 |      | PE3 |   | PE4 |      | PE5 |   | PE6 |
360	     +-----+   +-----+      +-----+   +-----+      +-----+   +-----+
361	        \         /            \         /            \         /
362	         \ ESI-1 /              \ ESI-2 /              \ ESI-3 /
363	          \     /                \     /                \     /
364	          +\---/+                +\---/+                +\---/+
365	          | \ / |                | \ / |                | \ / |
366	          +--+--+                +--+--+                +--+--+
367	             |                      |                      |
368	        Server-M1              Server-M2              Server-M3
369	             |                      |                      |
370	      VM-IP1, VM-IP2         VM-IP3, VM-IP4         VM-IP5, VM-IP6
371
372	                                  Figure 1
373
374	   As an example, consider a topology shown in Figure 1, with host VMs
375	   sharing the physical server MAC.  In steady state, IP1-M1 route is
376	   learnt at PE1, PE2 and advertised to remote PEs with a sequence
377	   number N.  Now, VM-IP1 is moved to MAC Server-M2.  ARP or ND based
378	   local learning at PE3, PE4 would now result in a new IP1-M2 route
379	   being learnt.  If route IP1-M2 is learnt as a new MAC+IP route and
380	   assigned a new sequence number of say 0, mobility procedure for VM-
381	   IP1 will not trigger across the overlay network.
382
383	   A sequence number assignment procedure needs to be defined to
384	   unambiguously determine the most recent IP reachability, IP to MAC
385	   binding, and MAC reachability for such a MAC sharing scenario.

[re-edit]
3.2.1. Host MAC+IP Move
This is the baseline scenario where a host move results in both the host's MAC and IP addresses moving together without altering the MAC-IP binding. The existing MAC mobility procedures defined in [RFC7432] can be leveraged to support this MAC+IP mobility scenario.

3.2.2. Host IP Move to a New MAC
This scenario involves a host move where the host's IP address is reassigned to a new MAC address.

3.2.2.1. VM Reload
A host reload or orchestrated move may cause a host to be re-spawned at a new location, resulting in a new MAC assignment while retaining the existing IP address. This results in the host's IP moving to a new MAC binding, as shown below:

IP-a, MAC-a ---> IP-a, MAC-b

3.2.2.2. MAC Sharing
This scenario considers cases where multiple hosts, each with a unique IP address, share a common MAC address. A host move results in a new MAC binding for the host IP. For example, hosts running on a single physical server might share the same MAC. Alternatively, an L2 access network behind a firewall may have all host IPs learned with a common firewall MAC. In these "shared MAC" scenarios, multiple local MAC-IP ARP entries may be learned with the same MAC. A host IP move to a new physical server could result in a new MAC association for the host IP.

3.2.2.3. Problem
In the aforementioned scenarios, a combined MAC+IP EVPN RT-2 advertised with a single sequence number attribute assumes a fixed IP-to-MAC mapping. A host IP move to a new MAC breaks this assumption and results in a new MAC+IP route. If this new route is independently assigned a new sequence number, the sequence number can no longer determine the most recent host IP reachability in a symmetric EVPN-IRB design or the most recent IP-to-MAC binding in an asymmetric EVPN-IRB design.

	                        +------------------------+
	                        | Underlay Network Fabric|
	                        +------------------------+

	     +-----+   +-----+      +-----+   +-----+      +-----+   +-----+
	     | PE1 |   | PE2 |      | PE3 |   | PE4 |      | PE5 |   | PE6 |
	     +-----+   +-----+      +-----+   +-----+      +-----+   +-----+
	        \         /            \         /            \         /
	         \ ESI-1 /              \ ESI-2 /              \ ESI-3 /
	          \     /                \     /                \     /
	          +\---/+                +\---/+                +\---/+
	          | \ / |                | \ / |                | \ / |
	          +--+--+                +--+--+                +--+--+
	             |                      |                      |
	        Server-M1              Server-M2              Server-M3
	             |                      |                      |
	      VM-IP1, VM-IP2         VM-IP3, VM-IP4         VM-IP5, VM-IP6

	                                  Figure 1

Figure 1 illustrates a topology with host VMs sharing the physical server MAC. In steady state, the IP1-M1 route is learned at PE1 and PE2 and advertised to remote PEs with a sequence number N. If VM-IP1 moves to Server-M2, ARP or ND-based local learning at PE3 and PE4 would result in a new IP1-M2 route. If this new route is assigned a sequence number of 0, the mobility procedure for VM-IP1 will not trigger across the overlay network.

A sequence number assignment procedure must be defined to unambiguously determine the most recent IP reachability, IP-to-MAC binding, and MAC reachability for such MAC sharing scenarios.

387	3.2.3.  Host MAC move to new IP
388
389	   This is a scenario where a host move or re-provisioning behind a new
390	   gateway location may result in the host getting a new IP address
391	   assigned, while keeping the same MAC.
392
393	3.2.3.1.  Problem
394
395	   The complication with this scenario is that MAC reachability could be
396	   carried via a combined MAC+IP route while a MAC only route may not be
397	   advertised at all.  A single sequence number association with the
398	   MAC+IP route again implicitly assumes a fixed mapping between MAC and
399	   IP.  A MAC move resulting in a new IP association for the host MAC
400	   breaks this assumption and results in a new MAC+IP route.  If this
401	   new MAC+IP route independently assumes a new sequence number, this
402	   mobility attribute can no longer be used to determine the most recent
403	   host MAC reachability.
404
405	                        +------------------------+
406	                        | Underlay Network Fabric|
407	                        +------------------------+
408	     +-----+   +-----+      +-----+   +-----+      +-----+   +-----+
409	     | PE1 |   | PE2 |      | PE3 |   | PE4 |      | PE5 |   | PE6 |
410	     +-----+   +-----+      +-----+   +-----+      +-----+   +-----+
411	        \         /            \         /            \         /
412	         \ ESI-1 /              \ ESI-2 /              \ ESI-3 /
413	          \     /                \     /                \     /
414	          +\---/+                +\---/+                +\---/+
415	          | \ / |                | \ / |                | \ / |
416	          +--+--+                +--+--+                +--+--+
417	             |                      |                      |
418	          Server1                Server2                Server3
419	             |                      |                      |
420	    VM-IP1-M1, VM-IP2-M2   VM-IP3-M3, VM-IP4-M4   VM-IP5-M5, VM-IP6-M6
421	                                  Figure 2
422
423	   As an example, consider a host VM IP1-M1 that is learnt locally at
424	   PE1, PE2 and advertised to remote hosts with a sequence number N.
425	   Consider a scenario where this VM with MAC M1 is re-provisioned at
426	   server 2, however, as part of this re-provisioning, assigned a
427	   different IP address say IP7.  IP7-M1 is learnt as a new route at
428	   PE3, PE4 and advertised to remote PEs with a sequence number of 0.
429	   As a result, L3 reachability to IP7 would be established across the
430	   overlay, however, MAC mobility procedure for M1 will not trigger as a
431	   result of this MAC-IP route advertisement.  If an optional MAC only
432	   route is also advertised, sequence number associated with the MAC
433	   only route would trigger MAC mobility as per [RFC7432].  However, in
434	   the absence of an additional MAC only route advertisement, a single
435	   sequence number advertised with a combined MAC+IP route may not be
436	   sufficient to update MAC reachability across the overlay.
437
438	   A MAC-IP sequence number assignment procedure needs to be defined to
439	   unambiguously determine the most recent MAC reachability in such a
440	   scenario without a MAC only route being advertised.
441
442	   Further, PE1/PE2, on learning new reachability for IP7-M1 via PE3/PE4
443	   MUST probe and delete any local IPs associated with MAC M1, such as
444	   IP1-M1 in the above example.
445
446	   Arguably, MAC mobility sequence number defined in [RFC7432], could be
447	   interpreted to apply only to the MAC part of MAC-IP route, and would
448	   hence cover this scenario.  This interpretation could be considered a
449	   clarification to [RFC7432] and one of the reasons for the common
450	   sequence number assignment procedure across all MAC-IP mobility
451	   scenarios detailed in this document.

[re-edit]
3.2.3.  Host MAC move to new IP
The complication in this scenario arises because MAC reachability can be carried via a combined MAC+IP route, whereas a MAC-only route may not be advertised. Associating a single sequence number with the MAC+IP route implicitly assumes a fixed MAC-to-IP mapping. A MAC move that results in a new IP association breaks this assumption and creates a new MAC+IP route. If this new route independently receives a new sequence number, the sequence number can no longer reliably indicate the most recent host MAC reachability.

	                        +------------------------+
	                        | Underlay Network Fabric|
	                        +------------------------+
	     +-----+   +-----+      +-----+   +-----+      +-----+   +-----+
	     | PE1 |   | PE2 |      | PE3 |   | PE4 |      | PE5 |   | PE6 |
	     +-----+   +-----+      +-----+   +-----+      +-----+   +-----+
	        \         /            \         /            \         /
	         \ ESI-1 /              \ ESI-2 /              \ ESI-3 /
	          \     /                \     /                \     /
	          +\---/+                +\---/+                +\---/+
	          | \ / |                | \ / |                | \ / |
	          +--+--+                +--+--+                +--+--+
	             |                      |                      |
	          Server1                Server2                Server3
	             |                      |                      |
	    VM-IP1-M1, VM-IP2-M2   VM-IP3-M3, VM-IP4-M4   VM-IP5-M5, VM-IP6-M6
	                                  Figure 2

For instance, consider host VM IP1-M1 learned locally at PE1 and PE2 and advertised to remote hosts with sequence number N. If this VM with MAC M1 is re-provisioned at Server2 and assigned a different IP address (e.g., IP7), the new IP7-M1 route learned at PE3 and PE4 would be advertised with sequence number 0. Consequently, L3 reachability to IP7 would be established across the overlay, but the MAC mobility procedure for M1 would not trigger due to the new MAC-IP route advertisement. Advertising an optional MAC-only route with its sequence number would trigger MAC mobility per [RFC7432]. However, without this additional advertisement, a single sequence number associated with a combined MAC+IP route may be insufficient to update MAC reachability across the overlay.

A MAC-IP sequence number assignment procedure is required to unambiguously determine the most recent MAC reachability in such scenarios without advertising a MAC-only route.

Furthermore, PE1 and PE2, upon learning new reachability for IP7-M1 via PE3 and PE4, must probe and delete any local IPs associated with MAC M1, such as IP1-M1.

It could be argued that the MAC mobility sequence number defined in [RFC7432] applies only to the MAC part of a MAC-IP route, thus covering this scenario. This interpretation could serve as a clarification to [RFC7432] and supports the need for a common sequence number assignment procedure across all MAC-IP mobility scenarios detailed in this document.

453	3.3.  EVPN All Active multi-homed ES
454	                           +------------------------+
455	                           | Underlay Network Fabric|
456	                           +------------------------+
457
458	                               +-----+   +-----+
459	                               | PE1 |   | PE2 |
460	                               +-----+   +-----+
461	                                 \\         //
462	                                  \\ ESI-1 //
463	                                   \\     /X
464	                                   +\\---//+
465	                                   | \\ // |
466	                                   +---+---+
467	                                       |
468	                                      CE1
469
470	                                  Figure 3
471
472	   Consider an EVPN-IRB overlay network shown in Figure 2, with hosts
473	   multi-homed to two or more PE devices via an all-active multi-homed
474	   ES.  MAC and ARP entries learnt on a local ES may also be
475	   synchronized across the multi-homing PE devices sharing this ES.
476	   This MAC and ARP SYNC enables local switching of intra and inter
477	   subnet ECMP traffic flows from remote hosts.  In other words, local
478	   MAC and ARP entries on a given ES may be learnt via local learning
479	   and / or via sync from another PE device sharing the same ES.
480
481	   For a host that is multi-homed to multiple PE devices via an all-
482	   active ES interface, local learning of host MAC and MAC-IP at each PE
483	   device is an independent asynchronous event, that is dependent on
484	   traffic flow and or ARP / ND response from the host hashing to a
485	   directly connected PE on the MC-LAG interface.  As a result, sequence
486	   number mobility attribute value assigned to a locally learnt MAC or
487	   MAC-IP route at each device may not always be the same, depending on
488	   transient states on the device at the time of local learning.
489
490	   As an example, consider a host VM that is deleted from ESI-2 and
491	   moved to ESI-1.  It is possible for host to be learnt on PE1
492	   following deletion of the remote route from PE3, PE4, while being
493	   learnt on PE2 prior to deletion of remote route from PE3, PE4.  If
494	   so, PE1 would process local host route learning as a new route and
495	   assign a sequence number of 0, while PE2 would process local host
496	   route learning as a remote to local move and assign a sequence number
497	   of N+1, N being the existing sequence number assigned at PE3, PE4.
498
499	   Inconsistent sequence numbers advertised from multi-homing devices
500	   introduces:
501
502	   *  Ambiguity with respect to how the remote PEs should handle paths
503	      with same ESI and different sequence numbers.  A remote PE might
504	      not program ECMP paths if it receives routes with different
505	      sequence numbers from a set of multi-homing PEs sharing the same
506	      ESI.
507
508	   *  Breaks consistent route versioning across the network overlay that
509	      is needed for EVPN mobility procedures to work.
510
511	   As an example, in this inconsistent state, PE2 would drop a remote
512	   route received for the same host with sequence number N (as its local
513	   sequence number is N+1), while PE1 would install it as the best route
514	   (as its local sequence number is 0).
515
516	   There is need for a mechanism to ensure consistency of sequence
517	   numbers advertised from a set of multi-homing devices for EVPN
518	   mobility to work reliably.
519
520	   In order to support mobility for multi-homed hosts using the sequence
521	   number mobility attribute, local MAC and MAC-IP routes learnt on a
522	   multi-homed ES MUST be advertised with the same sequence number by
523	   all PE devices that the ES is multi-homed to.  There is need for a
524	   mechanism to ensure consistency of sequence numbers assigned across
525	   these PEs.

[major]
* The text talks about PE3 and PE4 and about ESI-2, but the figure does not show this.
Can figure be corrected to show these components?
This will make it more clear how inconsistency with sequence numbers manifests.

[minor]
unsure why in thi informational section in the last paragraph uppercase MUST is used. BCP14 language does not apply to informational textblobs

[re-edit]
3.3.  EVPN All Active multi-homed ES

                           +------------------------+
                           | Underlay Network Fabric|
                           +------------------------+

                               +-----+   +-----+
                               | PE1 |   | PE2 |
                               +-----+   +-----+
                                 \\         //
                                  \\ ESI-1 //
                                   \\     /X
                                   +\\---//+
                                   | \\ // |
                                   +---+---+
                                       |
                                      CE1

                                  Figure 3

Consider an EVPN-IRB overlay network illustrated in Figure 3, where hosts are multi-homed to two or more PE devices via an all-active multi-homed ES. MAC and ARP entries learned on a local ES may also be synchronized across the multi-homing PE devices sharing this ES. This synchronization enables local switching of intra- and inter-subnet ECMP traffic flows from remote hosts. Thus, local MAC and ARP entries on a given ES may be learned through local learning and/or synchronization from another PE device sharing the same ES.

For a host that is multi-homed to multiple PE devices via an all-active ES interface, the local learning of host MAC and MAC-IP at each PE device is an independent asynchronous event, dependent on traffic flow or ARP/ND response from the host hashing to a directly connected PE on the MC-LAG interface. Consequently, the sequence number mobility attribute value assigned to a locally learned MAC or MAC-IP route at each device may not always be the same, depending on transient states on the device at the time of local learning.

For example, consider a host VM that is deleted from ESI-2 and moved to ESI-1. It is possible for the host to be learned on PE1 following the deletion of the remote route from PE3 and PE4, while being learned on PE2 prior to the deletion of the remote route from PE3 and PE4. In this case, PE1 would process local host route learning as a new route and assign a sequence number of 0, while PE2 would process local host route learning as a remote-to-local move and assign a sequence number of N+1, where N is the existing sequence number assigned at PE3 and PE4.

Inconsistent sequence numbers advertised from multi-homing devices introduce:

* Ambiguity regarding how remote PEs should handle paths with the same ESI but different sequence numbers. A remote PE might not program ECMP paths if it receives routes with different sequence numbers from a set of multi-homing PEs sharing the same ESI.
* Disruption of consistent route versioning across the network overlay, which is necessary for EVPN mobility procedures to function correctly.

For instance, in this inconsistent state, PE2 would drop a remote route received for the same host with sequence number N (since its local sequence number is N+1), while PE1 would install it as the best route (since its local sequence number is 0).

To support mobility for multi-homed hosts using the sequence number mobility attribute, local MAC and MAC-IP routes learned on a multi-homed ES must be advertised with the same sequence number by all PE devices to which the ES is multi-homed. There is a need for a mechanism to ensure the consistency of sequence numbers assigned across these PEs.

527	4.  Design Considerations
528
529	   To summarize, sequence number assignment scheme and implementation
530	   must take following considerations into account:
531
532	   *  MAC+IP may be learnt on an ES multi-homed to multiple PE devices,
533	      hence requires sequence numbers to be synchronized across multi-
534	      homing PE devices.
535
536	   *  MAC only RT-2 is optional in an IRB scenario and may not
537	      necessarily be advertised in addition to MAC+IP RT-2.
538
539	   *  A single MAC may be associated with multiple IPs, i.e., multiple
540	      host IPs may share a common MAC.
541
542	   *  A host IP move could result in host moving to a new MAC, resulting
543	      in a new IP to MAC association and a new MAC+IP route.
544
545	   *  A host MAC move to a new location could result in host MAC being
546	      associated with a different IP address, resulting in a new MAC to
547	      IP association and a new MAC+IP route.
548
549	   *  Local MAC-IP learn via ARP would always accompanied by a local MAC
550	      learn event resulting from the ARP packet.  MAC and MAC-IP
551	      learning, however, could happen in any order.
552
553	   *  Use cases discussed earlier that do not maintain a constant 1:1
554	      MAC-IP mapping across moves could potentially be addressed by
555	      using separate sequence numbers associated with MAC and IP
556	      components of MAC+IP route.  Maintaining two separate sequence
557	      numbers however adds significant overhead with respect to
558	      complexity, debugability, and backward compatibility.  Hence, this
559	      document addresses these requirements via a single sequence number
560	      attribute.

[re-edit]
To summarize, the sequence number assignment scheme and implementation must consider the following:

* Synchronization Across Multi-Homing PE Devices: MAC+IP may be learned on an ES multi-homed to multiple PE devices, requiring synchronized sequence numbers across these devices.

* Optional MAC-Only RT-2: In an IRB scenario, MAC-only RT-2 is optional and may not be advertised alongside MAC+IP RT-2.

* Multiple IPs Associated with a Single MAC: A single MAC may be linked to multiple IP addresses, indicating multiple host IPs sharing a common MAC.

* Host IP Movement: A host IP move may result in a new MAC association, necessitating a new IP to MAC association and a new MAC+IP route.

* Host MAC Movement: A host MAC move may result in a new IP association, requiring a new MAC to IP association and a new MAC+IP route.

* Local MAC-IP Learning via ARP: Local MAC-IP learning via ARP always accompanies a local MAC learning event resulting from the ARP packet. However, MAC and MAC-IP learning can occur in any order.

* Separate Sequence Numbers for MAC and IP: Use cases that do not maintain a constant 1:1 MAC-IP mapping across moves could potentially be addressed by using separate sequence numbers for MAC and IP components of the MAC+IP route. However, maintaining two separate sequence numbers adds significant complexity, debugging challenges, and backward compatibility issues. Therefore, this document addresses these requirements using a single sequence number attribute.

562	5.  Solution Components
563
564	   This section goes over the main components of the EVPN IRB mobility
565	   solution specified in this document.  Later sections will specify
566	   exact sequence number assignment procedures resulting from concepts
567	   described in this section.
568
569	5.1.  Sequence Number Inheritance
570
571	   The main idea presented here is to view a local MAC-IP route as a
572	   child of the corresponding local MAC route within the local context
573	   of a PE, such that the local MAC-IP route inherits the sequence
574	   number attribute from the parent local MAC only route:
575
576	   Mx-IPx -----> Mx (seq# = N)
578
578	   As a result, both parent MAC and child MAC-IP routes share one common
579	   sequence number associated with the parent MAC route.  Doing so
580	   ensures that a single sequence number attribute carried in a combined
581	   MAC+IP route represents sequence number for both a MAC only route as
582	   well as a MAC+IP route, and hence makes advertisement of the MAC only
583	   route truly optional.  As a result, optional MAC only route with its
584	   own sequence number is not required to establish the most recent
585	   reachability for a MAC in the overlay network.  Specifically, this
586	   enables a MAC to assume a different IP address on a move, and still
587	   be able to establish the most recent reachability to the MAC across
588	   the overlay network via the mobility attribute associated with the
589	   MAC+IP route advertisement.  As an example, when Mx moves to a new
590	   location, it would result in local Mx being assigned a higher
591	   sequence number at its new location as per [RFC7432].  If this move
592	   results in Mx assuming a different IP address, IPz, local Mx+IPz
593	   route would inherit the new sequence number from Mx.
594
595	   Local MAC and local MAC-IP routes would typically be sourced from
596	   data plane learning and ARP learning respectively, and could get
597	   learnt in the control plane in any order.  Implementation could
598	   either replicate the inherited sequence number in each MAC-IP entry
599	   OR maintain a single attribute in the parent MAC by creating a
600	   forward reference local MAC object for cases where a local MAC-IP is
601	   learnt before the local MAC.
602
603	5.2.  MAC Sharing

605	   Further, for the shared MAC scenario, this results in multiple local
606	   MAC-IP siblings inheriting a sequence number attribute from the
607	   common parent MAC route:
608
609	     Mx-IP1 -----
610	      |          |
611	     Mx-IP2 -----
612	       .         |
613	       .         +---> Mx (seq# = N)
614	       .         |
615	     Mx-IPw -----
616	       |         |
617	     Mx-IPx -----
627
619	                                  Figure 4
620
621	   In such a case, a host-IP move to a different physical server would
622	   result in IP moving to a new MAC binding.  A new MAC-IP route
623	   resulting from this move must now be advertised with a sequence
624	   number that is higher than the previous MAC-IP route for this IP,
625	   advertised from the prior location.  As an example, consider a route
626	   Mx-IPx that is currently advertised with sequence number N from PE1.
627	   IPx moving to a new physical server behind PE2 results in IPx being
628	   associated with MAC Mz.  A new local Mz-IPx route resulting from this
629	   move at PE2 must now be advertised with a sequence number higher than
630	   N and higher than the previous Mz sequence number M.  This is so that
631	   PE devices, including PE1, PE2, and other remote PE devices that are
632	   part of the overlay can clearly determine and program the most recent
633	   MAC binding and reachability for the IP.  PE1, on receiving this new
634	   Mz-IPx route with sequence number say, N+1, for symmetric IRB case,
635	   would update IPx reachability via PE2 in forwarding, for asymmetric
636	   IRB case, would update IPx's ARP binding to Mz.  In addition, PE1
637	   would clear and withdraw the stale Mx-IPx route with the lower
638	   sequence number.
639
640	   This also implies that sequence number associated with local MAC Mz
641	   and all local MAC-IP children of Mz at PE2 must now be incremented to
642	   N+1 or to M+1 if the previous Mz sequence number M is greater than N,
643	   and re-advertised across the overlay.  While this re-advertisement of
644	   all local MAC-IP children routes affected by the parent MAC route is
645	   an overhead, it avoids the need for two separate sequence number
646	   attributes to be maintained and advertised for IP and MAC components
647	   of MAC+IP RT-2.  Implementation would need to be able to lookup MAC-
648	   IP routes for a given IP and update sequence number for it's parent
649	   MAC and its MAC-IP children.
650
651	5.3.  Multi-homing Mobility Synchronization
652
653	   In order to support mobility for multi-homed hosts, local MAC and
654	   MAC-IP routes learnt on a shared ES MUST be advertised with the same
655	   sequence number by all PE devices that the ES is multi-homed to.
656	   This also applies to local MAC only routes. local MAC and MAC-IP may
657	   be learnt natively via data plane and ARP/ND respectively as well as
658	   via SYNC from another multi-homing PE to achieve local switching.
659	   Local and SYNC route learning can happen in any order.  Local MAC-IP
660	   routes advertised by all multi-homing PE devices sharing the ES must
661	   carry the same sequence number, independent of the order in which
662	   they are learnt.  This implies:
663
664	   *  On local or SYNC MAC-IP route learning, sequence number for the
665	      local MAC-IP route MUST be compared and updated to the higher
666	      value.
667
668	   *  On local or SYNC MAC route learning, sequence number for the local
669	      MAC route MUST be compared and updated to the higher value.
670
671	   If an update to local MAC-IP sequence number is required as a result
672	   of the above comparison with SYNC MAC-IP route, it would essentially
673	   amount to a sequence number update on the parent local MAC, resulting
674	   in inherited sequence number update on the MAC-IP route.

[major]
* is the arrow used in the small figure correct? Should it not be the other way around if the sequence number is inherited? w.o.w. Mx (seq# = N) -----> Mx-IPx ?
* similar with the other figure in section 5.2

[re-edit]
5. Solution Components
This section outlines the main components of the EVPN IRB mobility solution specified in this document. Subsequent sections will detail the exact sequence number assignment procedures based on the concepts described here.

5.1. Sequence Number Inheritance
The key concept presented here is to treat a local MAC-IP route as a child of the corresponding local MAC route within the local context of a PE. This ensures that the local MAC-IP route inherits the sequence number attribute from the parent local MAC-only route:

	   Mx-IPx -----> Mx (seq# = N)

Thus, both the parent MAC and child MAC-IP routes share a common sequence number associated with the parent MAC route. This ensures that a single sequence number attribute carried in a combined MAC+IP route represents the sequence number for both a MAC-only route and a MAC+IP route, making the advertisement of the MAC-only route truly optional. This enables a MAC to assume a different IP address upon moving and still establish the most recent reachability to the MAC across the overlay network via the mobility attribute associated with the MAC+IP route advertisement. For instance, when Mx moves to a new location, it would be assigned a higher sequence number at its new location per [RFC7432]. If this move results in Mx assuming a different IP address, IPz, the local Mx+IPz route would inherit the new sequence number from Mx.

Local MAC and local MAC-IP routes are typically sourced from data plane learning and ARP learning, respectively, and can be learned in the control plane in any order. Implementation can either replicate the inherited sequence number in each MAC-IP entry or maintain a single attribute in the parent MAC by creating a forward reference local MAC object for cases where a local MAC-IP is learned before the local MAC.

5.2. MAC Sharing
For the shared MAC scenario, multiple local MAC-IP siblings inherit the sequence number attribute from the common parent MAC route:

 Mx-IP1 -----
  |          |
 Mx-IP2 -----
   .         |
   .         +---> Mx (seq# = N)
   .         |
 Mx-IPw -----
   |         |
 Mx-IPx -----

In such cases, a host-IP move to a different physical server results in the IP moving to a new MAC binding. A new MAC-IP route resulting from this move must be advertised with a sequence number higher than the previous MAC-IP route for this IP, advertised from the prior location. For example, consider a route Mx-IPx currently advertised with sequence number N from PE1. If IPx moves to a new physical server behind PE2 and is associated with MAC Mz, the new local Mz-IPx route must be advertised with a sequence number higher than N and the previous Mz sequence number M. This allows PE devices, including PE1, PE2, and other remote PE devices, to determine and program the most recent MAC binding and reachability for the IP. PE1, upon receiving this new Mz-IPx route with sequence number N+1, would update IPx reachability via PE2 for symmetric IRB and update IPx's ARP binding to Mz for asymmetric IRB, while clearing and withdrawing the stale Mx-IPx route with the lower sequence number.

This implies that the sequence number associated with local MAC Mz and all local MAC-IP children of Mz at PE2 must be incremented to N+1 or M+1 if the previous Mz sequence number M is greater than N and re-advertised across the overlay. While this re-advertisement of all local MAC-IP children routes affected by the parent MAC route adds overhead, it avoids the need for maintaining and advertising two separate sequence number attributes for IP and MAC components of MAC+IP RT-2. Implementation must be able to look up MAC-IP routes for a given IP and update the sequence number for its parent MAC and its MAC-IP children.

5.3. Multi-Homing Mobility Synchronization
To support mobility for multi-homed hosts, local MAC and MAC-IP routes learned on a shared ES must be advertised with the same sequence number by all PE devices to which the ES is multi-homed. This applies to local MAC-only routes as well. Local MAC and MAC-IP may be learned natively via data plane and ARP/ND respectively, as well as via SYNC from another multi-homing PE to achieve local switching. Local and SYNC route learning can occur in any order. Local MAC-IP routes advertised by all multi-homing PE devices sharing the ES must carry the same sequence number, independent of the order in which they are learned. This implies:

* On local or SYNC MAC-IP route learning, the sequence number for the local MAC-IP route must be compared and updated to the higher value.

* On local or SYNC MAC route learning, the sequence number for the local MAC route must be compared and updated to the higher value.

If an update to the local MAC-IP sequence number is required as a result of the comparison with the SYNC MAC-IP route, it essentially amounts to a sequence number update on the parent local MAC, resulting in an inherited sequence number update on the MAC-IP route.

676	6.  Requirements for Sequence Number Assignment
677
678	   Following sections specify sequence number assignment procedure
679	   needed on local and SYNC MAC and MAC-IP route learning events in
680	   order to accomplish the above.
681
682	6.1.  Local MAC-IP learning
683
684	   A local Mx-IPx learning via ARP or ND should result in computation OR
685	   re-computation of the parent MAC Mx's sequence number, following
686	   which the MAC-IP route Mx-IPx would simply inherit parent MAC's
687	   sequence number.  The parent MAC Mx Sequence number MUST be computed
688	   as follows:
689
690	   *  MUST be higher than any existing remote MAC route for Mx, as per
691	      [RFC7432].
692
693	   *  MUST be at least equal to corresponding SYNC MAC sequence number
694	      if one is present.
695
696	   *  If the IP is also associated with a different remote MAC "Mz",
697	      MUST be higher than the "Mz" sequence number.
698
699	   Once the new sequence number for MAC route Mx is computed as per
700	   above, all local MAC-IPs associated with MAC Mx MUST inherit the
701	   updated sequence number.
702
703	6.2.  Local MAC learning
704
705	   The local MAC Mx Sequence number MUST be computed as follows:
705
707	   *  MUST be higher than any existing remote MAC route for Mx, as per
708	      [RFC7432].
709
710	   *  MUST be at least equal to the corresponding SYNC MAC sequence
711	      number if one is present.
712
713	   *  Once the new sequence number for MAC route Mx is computed as per
714	      above, all local MAC-IPs associated with MAC Mx MUST inherit the
715	      updated sequence number.
716
717	   Note that the local MAC sequence number might already be present if
718	   there was a local MAC-IP learnt prior to the local MAC, in which case
719	   the above may not result in any change in local MAC's sequence
720	   number.
721
722	6.3.  Remote MAC or MAC-IP Update
723
724	   On receiving a remote MAC OR MAC-IP route update associated with a
725	   MAC Mx with a sequence number that is
726
727	   *  either higher than the sequence number assigned to a local route
728	      for MAC Mx,
729
730	   *  or equal to the sequence number assigned to a local route for MAC
731	      Mx, but the remote route is selected as best because of lower VTEP
732	      IP as per [RFC7432],
733
734	   following handling IS REQUIRED on the receiving PE:
735
736	   *  the PE MUST trigger probe and deletion procedure for all local IPs
737	      associated with MAC Mx.
738
739	   *  the PE MUST trigger deletion procedure for local MAC route for Mx.
740
741	6.4.  REMOTE (SYNC) MAC update
742
743	   On receiving a REMOTE SYNC, the corresponding local MAC Mx (if
744	   present) sequence number should be re- computed as follows:
745
746	   *  If the current sequence number is less than the received SYNC MAC
747	      sequence number, it MUST be increased to be equal to received SYNC
748	      MAC sequence number.
749
750	   *  If a local MAC sequence number is updated as a result of the
751	      above, all local MAC-IPs associated with MAC Mx MUST inherit the
752	      updated sequence number.
753
754	6.5.  REMOTE (SYNC) MAC-IP update
755
756	   Receiving a SYNC MAC-IP for a locally attached host results in a
757	   derived SYNC MAC Mx route entry, as MAC only RT-2 advertisement is
758	   optional.  The corresponding local MAC Mx (if present) sequence
759	   number should be re-computed as follows:
760
761	   *  If the current sequence number is less than the received SYNC MAC
762	      sequence number, it MUST be increased to be equal to received SYNC
763	      MAC sequence number.
764
765	   *  If a local MAC sequence number is updated as a result of the
766	      above, all local MAC-IPs associated with MAC Mx MUST inherit the
767	      updated sequence number.
768
769	6.6.  Interoperability
770
771	   In general, if all PE nodes in the overlay network follow the above
772	   sequence number assignment procedures, and the PE is advertising both
773	   MAC+IP and MAC routes, sequence numbers advertised with the MAC and
774	   MAC+IP routes with the same MAC would always be the same.  However,
775	   an inter-op scenario with a different implementation could arise,
776	   where a PE implementation non-compliant with this document or with
777	   [RFC7432] assigns and advertises independent sequence numbers to MAC
778	   and MAC+IP routes.  To handle this case, if different sequence
779	   numbers are received for remote MAC+IP and corresponding remote MAC
780	   routes from a remote PE, sequence number associated with the remote
781	   MAC route MUST be computed as:
782
783	   *  Highest of all the received sequence numbers with remote MAC+IP
784	      and MAC routes with the same MAC.
785
786	   *  MAC sequence number would be re-computed on a MAC or MAC+IP route
787	      withdraw as per above.
788
789	   A MAC and / or IP move to the local PE would now result in the MAC
790	   (and hence all MAC-IP) sequence numbers being incremented from the
791	   above computed remote MAC sequence number.
792
793	   If MAC only routes are not advertised at all, and different sequence
794	   numbers are received with multiple MAC+IP routes for a given MAC, the
795	   sequence number associated with the derived remote MAC route should
796	   still be computed as the highest of all of the received MAC+IP
797	   sequence numbers with the same MAC.
798
799	6.7.  MAC Sharing Race Condition
800
801	   In a MAC sharing use case described in section 5.2, a race condition
802	   is possible with simultaneous host moves between a pair of PEs.  As
803	   an example, consider PE1 with local host IPs I1 and I2 sharing MAC
804	   M1, and PE2 with local host IPs I3 and I4 sharing MAC M2.  A
805	   simultaneous move of I1 from PE1 to PE2 and of I3 from PE2 to PE1,
806	   such that I3 is learnt on PE1 before I1's local entry has been probed
807	   out on PE1 and/or I1 is learnt on PE2 before I3's local entry has
808	   been probed out on PE2 may trigger a race condition.  This race
809	   condition together with MAC sequence number assignment rules defined
810	   in section 6.1 can cause new mac-ip routes I1-M2 and I3-M1 to bounce
811	   a couple of times with an incremented sequence number until stale
812	   entries I1-M1 and I3-M2 have been probed out from PE1 and PE2
813	   respectively.  An implementation MUST ensure proper probing
814	   procedures to remove stale ARP, ND, and local MAC entries, following
815	   a move, on learning remote routes as defined in section 6.3 (and as
816	   per [RFC9135]) to minimize exposure to this race condition.
817
818	6.8.  Mobility Convergence
819
820	   This sections is optional and details ARP and ND probing procedures
821	   that MAY be implemented to achieve faster host re- learning and
822	   convergence on mobility events.
822
824	   *  Following a host move from PE1 to PE2, the host's MAC is
825	      discovered at PE2 as a local MAC via a data frames received from
826	      the host.  If PE2 has a prior remote MAC-IP host route for this
827	      MAC from PE1, an ARP/ND probe MAY be triggered at PE2 to learn the
828	      MAC-IP as a local adjacency and trigger EVPN RT-2 advertisement
829	      for this MAC-IP across the overlay with new reachability via PE2.
830	      This results in a reliable "event based" host IP learning
831	      triggered by a "MAC learning event" across the overlay, and hence
832	      faster convergence of overlay routed flows to the host.
833
834	   *  Following a host move from PE1 to PE2, once PE1 receives a MAC or
835	      MAC-IP route from PE2 with a higher sequence number, an ARP/ND
836	      probe MAY be triggered at PE1 to clear the stale local MAC-IP
837	      neighbor adjacency or to re-learn the local MAC-IP in case the
838	      host has moved back or is duplicate.
838
840	   *  Following a local MAC age-out, if there is a local IP adjacency
841	      with this MAC, an ARP/ND probe MAY be triggered for this IP to
842	      either re-learn the local MAC and maintain local l3 and l2
843	      reachability to this host or to clear the ARP/ND entry in case the
844	      host is indeed no longer local.  Note that this accomplishes
845	      clearance of stale ARP entries, triggered by a MAC age-out event
846	      even when the ARP refresh timer was longer than the MAC age-out
847	      timer.  Clearing of stale IP neighbor entries in-turn facilitates
848	      traffic convergence in the event that the host was silent and not
849	      discovered at its new location.  Once the stale neighbor entry for
850	      the host is cleared, routed traffic flow destined for the host can
851	      re-trigger ARP/ND discovery for this host at the new location.
852
853	6.8.1.  Generalized Probing Logic
854
855	   The above probing logic may be generalized as probing for an IP
856	   neighbor anytime a resolving parent MAC route is "inconsistent" with
857	   the MAC- IP neighbor route, where being inconsistent is defined as
858	   being not present or conflicting in terms of the route source being
859	   local OR remote.  The MAC-IP to MAC parent relationship described
860	   earlier in this document in section 5.1 MAY be used to achieve this
861	   logic.

[major]
* for my own understanding: in section 6.2 first bullet point, make me wonder if the connected ESI is share between two PEs. Would the requirement potentially lead to a count to infinity when two PEs connect to a shared ESI?

* section 6.6: How would an implementation detect that the remote implementation does not support the behavior? Could that be explicit explained in the text?

* section 6.7: THis section i did not understand. Too many moving parts. Can this be explained more explicit or elaborative?

* section 6.8: What network figure is referenced towards?

[re-edit]
6. Requirements for Sequence Number Assignment
The following sections specify the sequence number assignment procedures required for local and SYNC MAC and MAC-IP route learning events to achieve the objectives outlined.

6.1. Local MAC-IP Learning
A local Mx-IPx learning via ARP or ND should result in the computation or re-computation of the parent MAC Mx's sequence number, following which the MAC-IP route Mx-IPx inherits the parent MAC's sequence number. The parent MAC Mx sequence number MUST be computed as follows:

* MUST be higher than any existing remote MAC route for Mx, as per [RFC7432].

* MUST be at least equal to the corresponding SYNC MAC sequence number, if present.

* If the IP is also associated with a different remote MAC "Mz," it MUST be higher than the "Mz" sequence number.

Once the new sequence number for MAC route Mx is computed as per the above criteria, all local MAC-IPs associated with MAC Mx MUST inherit the updated sequence number.

6.2. Local MAC Learning
The local MAC Mx sequence number MUST be computed as follows:

* MUST be higher than any existing remote MAC route for Mx, as per [RFC7432].

* MUST be at least equal to the corresponding SYNC MAC sequence number, if present.

Once the new sequence number for MAC route Mx is computed as per the above criteria, all local MAC-IPs associated with MAC Mx MUST inherit the updated sequence number. Note that the local MAC sequence number might already be present if there was a local MAC-IP learned prior to the local MAC, in which case the above may not result in any change in the local MAC's sequence number.

6.3. Remote MAC or MAC-IP Update
Upon receiving a remote MAC or MAC-IP route update associated with a MAC Mx with a sequence number that is:

* Either higher than the sequence number assigned to a local route for MAC Mx,

* Or equal to the sequence number assigned to a local route for MAC Mx, but the remote route is selected as best due to a lower VTEP IP as per [RFC7432],

the following actions are REQUIRED on the receiving PE:

* The PE MUST trigger a probe and deletion procedure for all local IPs associated with MAC Mx.

* The PE MUST trigger a deletion procedure for the local MAC route for Mx.

6.4. REMOTE (SYNC) MAC Update
Upon receiving a REMOTE SYNC, the corresponding local MAC Mx (if present) sequence number should be re-computed as follows:

* If the current sequence number is less than the received SYNC MAC sequence number, it MUST be increased to be equal to the received SYNC MAC sequence number.

* If a local MAC sequence number is updated as a result of the above, all local MAC-IPs associated with MAC Mx MUST inherit the updated sequence number.

6.5. REMOTE (SYNC) MAC-IP Update
Receiving a SYNC MAC-IP for a locally attached host results in a derived SYNC MAC Mx route entry, as the MAC-only RT-2 advertisement is optional. The corresponding local MAC Mx (if present) sequence number should be re-computed as follows:

* If the current sequence number is less than the received SYNC MAC sequence number, it MUST be increased to be equal to the received SYNC MAC sequence number.

* If a local MAC sequence number is updated as a result of the above, all local MAC-IPs associated with MAC Mx MUST inherit the updated sequence number.

6.6. Interoperability
Generally, if all PE nodes in the overlay network follow the above sequence number assignment procedures and the PE is advertising both MAC+IP and MAC routes, the sequence numbers advertised with the MAC and MAC+IP routes with the same MAC would always be the same. However, an interoperability scenario with a different implementation could arise, where a non-compliant PE implementation assigns and advertises independent sequence numbers to MAC and MAC+IP routes. To handle this case, if different sequence numbers are received for remote MAC+IP and corresponding remote MAC routes from a remote PE, the sequence number associated with the remote MAC route MUST be computed as:

* The highest of all received sequence numbers with remote MAC+IP and MAC routes with the same MAC.

* The MAC sequence number would be re-computed on a MAC or MAC+IP route withdraw as per the above. 

A MAC and/or IP move to the local PE would then result in the MAC (and hence all MAC-IP) sequence numbers being incremented from the above computed remote MAC sequence number. 

If MAC-only routes are not advertised at all, and different sequence numbers are received with multiple MAC+IP routes for a given MAC, the sequence number associated with the derived remote MAC route should still be computed as the highest of all received MAC+IP sequence numbers with the same MAC.

6.7. MAC Sharing Race Condition 
*************************************************************
****This section i was not able to process and understand****
*************************************************************
In a MAC sharing use case described in section 5.2, a race condition
is possible with simultaneous host moves between a pair of PEs.  As
an example, consider PE1 with local host IPs I1 and I2 sharing MAC
M1, and PE2 with local host IPs I3 and I4 sharing MAC M2.  A
simultaneous move of I1 from PE1 to PE2 and of I3 from PE2 to PE1,
such that I3 is learnt on PE1 before I1's local entry has been probed
out on PE1 and/or I1 is learnt on PE2 before I3's local entry has
been probed out on PE2 may trigger a race condition.  This race
condition together with MAC sequence number assignment rules defined
in section 6.1 can cause new mac-ip routes I1-M2 and I3-M1 to bounce
a couple of times with an incremented sequence number until stale
entries I1-M1 and I3-M2 have been probed out from PE1 and PE2
respectively.  An implementation MUST ensure proper probing
procedures to remove stale ARP, ND, and local MAC entries, following
a move, on learning remote routes as defined in section 6.3 (and as
per [RFC9135]) to minimize exposure to this race condition.

6.8. Mobility Convergence
This section is optional and details ARP and ND probing procedures that MAY be implemented to achieve faster host re-learning and convergence on mobility events.

* Following a host move from PE1 to PE2, the host's MAC is discovered at PE2 as a local MAC via data frames received from the host. If PE2 has a prior remote MAC-IP host route for this MAC from PE1, an ARP/ND probe MAY be triggered at PE2 to learn the MAC-IP as a local adjacency and trigger EVPN RT-2 advertisement for this MAC-IP across the overlay with new reachability via PE2. This results in a reliable "event-based" host IP learning triggered by a "MAC learning event" across the overlay, and hence faster convergence of overlay routed flows to the host.

* Following a host move from PE1 to PE2, once PE1 receives a MAC or MAC-IP route from PE2 with a higher sequence number, an ARP/ND probe MAY be triggered at PE1 to clear the stale local MAC-IP neighbor adjacency or to re-learn the local MAC-IP in case the host has moved back or is duplicated.

* Following a local MAC age-out, if there is a local IP adjacency with this MAC, an ARP/ND probe MAY be triggered for this IP to either re-learn the local MAC and maintain local L3 and L2 reachability to this host or to clear the ARP/ND entry if the host is no longer local. This accomplishes the clearance of stale ARP entries triggered by a MAC age-out event even when the ARP refresh timer is longer than the MAC age-out timer. Clearing stale IP neighbor entries facilitates traffic convergence if the host was silent and not discovered at its new location. Once the stale neighbor entry for the host is cleared, routed traffic flow destined for the host can re-trigger ARP/ND discovery for this host at the new location.

6.8.1. Generalized Probing Logic
The above probing logic may be generalized as probing for an IP neighbor anytime a resolving parent MAC route is inconsistent with the MAC-IP neighbor route, where inconsistency is defined as being not present or conflicting in terms of the route source being local or remote. The MAC-IP to MAC parent relationship described in section 5.1 MAY be used to achieve this logic.

863	7.  Routed Overlay
864
865	   An additional use case is possible, such that traffic to an end host
866	   in the overlay is always IP routed.  In a purely routed overlay such
867	   as this:
868
869	   *  A host MAC is never advertised in the EVPN overlay control plane.
870
871	   *  Host /32 or /128 IP reachability is distributed across the overlay
872	      via EVPN route type 5 (RT-5) along with a zero or non- zero ESI.
873
874	   *  An overlay IP subnet may still be stretched across the underlay
875	      fabric, however, intra-subnet traffic across the stretched overlay
876	      is never bridged.
877
878	   *  Both inter-subnet and intra-subnet traffic, in the overlay is IP
879	      routed at the EVPN PE.
880
881	   Please refer to [RFC7814] for more details.
882
883	   Host mobility within the stretched subnet would still need to be
884	   supported for this use.  In the absence of any host MAC routes,
885	   sequence number mobility Extended Community specified in [RFC7432],
886	   section 7.7 may be associated with a /32 OR /128 host IP prefix
887	   advertised via EVPN route type 5.  MAC mobility procedures defined in
888	   [RFC7432] can now be applied as is to host IP prefixes:
889
890	   *  On local learning of a host IP, on a new ESI, the host IP MUST be
891	      advertised with a sequence number attribute that is higher than
892	      what is currently advertised with the old ESI.
893
894	   *  On receiving a host IP route advertisement with a higher sequence
895	      number, a PE MUST trigger ARP/ND probe and deletion procedures on
896	      any local route for that IP with a lower sequence number.  A PE
897	      would essentially move the forwarding entry to point to the remote
898	      route with a higher sequence number and send an ARP/ND PROBE for
899	      the local IP route.  If the IP has indeed moved, PROBE would
900	      timeout and the local IP host route would be deleted.
901
902	   Note that there is still only one sequence number associated with a
903	   host route at any time.  For earlier use cases where a host MAC is
904	   advertised along with the host IP, a sequence number is only
905	   associated with a MAC.  Only if the MAC is not advertised at all, as
906	   in this use case, is a sequence number associated with a host IP.
907
908	   Note that this mobility procedure would not apply to "anycast IPv6"
909	   hosts advertised via NA messages with 0-bit=0.  Please refer to
910	   [RFC9161].

[major]
* Unsure what purpose of 0-bit=0 is and where it is explained in RFC9161. Some explicit reference and explanation could help the draft 

[re-edit]
7. Routed Overlay
An additional use case involves traffic to an end host in the overlay being entirely IP routed. In such a purely routed overlay:

* A host MAC is never advertised in the EVPN overlay control plane.

* Host /32 or /128 IP reachability is distributed across the overlay via EVPN Route Type 5 (RT-5) along with a zero or non-zero ESI.

* An overlay IP subnet may still be stretched across the underlay fabric; however, intra-subnet traffic across the stretched overlay is never bridged.

* Both inter-subnet and intra-subnet traffic in the overlay is IP routed at the EVPN PE.

Refer to [RFC7814] for more details.

Host mobility within the stretched subnet still needs support. In the absence of host MAC routes, the sequence number mobility Extended Community specified in [RFC7432], section 7.7, MAY be associated with a /32 or /128 host IP prefix advertised via EVPN Route Type 5. MAC mobility procedures defined in [RFC7432] can be applied to host IP prefixes as follows:

* On local learning of a host IP on a new ESI, the host IP MUST be advertised with a sequence number higher than what is currently advertised with the old ESI.

* On receiving a host IP route advertisement with a higher sequence number, a PE MUST trigger ARP/ND probe and deletion procedures on any local route for that IP with a lower sequence number. The PE will update the forwarding entry to point to the remote route with a higher sequence number and send an ARP/ND probe for the local IP route. If the IP has moved, the probe will time out, and the local IP host route will be deleted.

Note that there is only one sequence number associated with a host route at any time. For previous use cases where a host MAC is advertised along with the host IP, a sequence number is only associated with the MAC. If the MAC is not advertised, as in this use case, a sequence number is associated with the host IP.

This mobility procedure does not apply to "anycast IPv6" hosts advertised via NA messages with 0-bit=0. Refer to [RFC9161] for more details.

912	8.  Duplicate Host Detection
913
914	   Duplicate host detection scenarios across EVPN IRB can be classified
915	   as follows:
916
917	   *  Scenario A: where two hosts have the same MAC (host IPs may or may
918	      not be duplicate).
919
920	   *  Scenario B: where two hosts have the same IP but different MACs.
920
922	   *  Scenario C: where two hosts have the same IP and host MAC is not
923	      advertised at all.
924
925	   Duplicate detection procedures for scenario B and C would not apply
926	   to "anycast IPv6" hosts advertised via NA messages with 0-bit=0.
927	   Please refer to [RFC9161].
928
929	8.1.  Scenario A
920
931	   For all use cases where duplicate hosts have the same MAC, the MAC is
932	   detected as duplicate via the duplicate MAC detection procedure
933	   described in [RFC7432].  Corresponding MAC-IP routes with the same
934	   MAC do not require duplicate detection and MUST simply inherit the
935	   duplicate property from the corresponding MAC route.  In other words,
936	   if a MAC route is in duplicate state, all corresponding MAC-IP routes
937	   MUST also be treated as duplicate.  Duplicate detection procedure
938	   need only be applied to MAC routes.
939
940	8.2.  Scenario B
941
942	   Due to misconfiguration, a situation may arise where hosts with
943	   different MACs are configured with the same IP.  This scenario would
944	   not be detected by [RFC7432] duplicate MAC detection procedures and
945	   would result in incorrect forwarding of routed traffic destined to
946	   this IP.
947
948	   Such a situation, on local MAC-IP learning, would be detected as a
949	   move scenario via the following local MAC sequence number computation
950	   procedure described earlier in section 6.1:
951
952	   *  If the IP is also associated with a different remote MAC "Mz",
953	      MUST be higher than "Mz" sequence number.
954
955	   Such a move that results in sequence number increment on local MAC
956	   because of a remote MAC-IP route associated with a different MAC MUST
957	   be counted as an "IP move" against the "IP" independent of the MAC.
958	   Duplicate detection procedure described in [RFC7432] can now be
959	   applied to an "IP" entity independent of MAC.  Once an IP is detected
960	   as duplicate, corresponding MAC-IP route should be treated as
961	   duplicate.  Associated MAC routes and any other MAC-IP routes
962	   associated with this MAC should not be affected.
963
964	8.2.1.  Duplicate IP Detection Procedure for Scenario B
065
966	   The duplicate IP detection procedure for such a scenario are
967	   specified in [RFC9161].  What counts as an "IP move" in this scenario
968	   is further clarified as follows:
969
970	   *  On learning a local MAC-IP route Mx-IPx, check if there is an
971	      existing remote or local route for IPx with a different MAC
972	      association, say, Mz-IPx.  If so, count this as an "IP move" count
973	      for IPx, independent of the MAC.
974
975	   *  On learning a remote MAC-IP route Mz-IPx, check if there is an
976	      existing local route for IPx with a different MAC association,
977	      say, Mx-IPx.  If so, count this as an "IP move" count for IPx,
978	      independent of the MAC.
979
980	   A MAC-IP route SHOULD be treated as duplicate if either of the
981	   following two conditions are met:
982
983	   *  The corresponding MAC route is marked as duplicate via existing
984	      duplicate detection procedure.
985
986	   *  The corresponding IP is marked as duplicate via extended procedure
987	      described above.
988
989	8.3.  Scenario C
990
991	   For a purely routed overlay scenario described in section 7, where
992	   only a host IP is advertised via EVPN RT-5, together with a sequence
993	   number mobility attribute, duplicate MAC detection procedures
994	   specified in [RFC7432] can be intuitively applied to IP only host
995	   routes for the purpose of duplicate IP detection.
996
997	   *  On learning a local host IP route IPx, check if there is an
998	      existing remote or local route for IPx with a different ESI
999	      association.  If so, count this as an "IP move" count for IPx.
1000
1001	   *  On learning a remote host IP route IPx, check if there is an
1002	      existing local route for IPx with a different ESI association.  If
1003	      so, count this as an "IP move" count for IPx.
1004
1005	   *  With configurable parameters "N" and "M", if "N" IP moves are
1006	      detected within "M" seconds for IPx, treat IPx as duplicate.
1007
1008	8.4.  Duplicate Host Recovery
1009
1010	   Once a MAC or IP is marked as duplicate and frozen, corrective action
1011	   must be taken to un-provision one of the duplicate MAC or IP.  Un-
1012	   provisioning a duplicate MAC or IP in this context refers to a
1013	   corrective action taken on the host side.  Once one of the duplicate
1014	   MAC or IP is un-provisioned, normal operation would not resume until
1015	   the duplicate MAC or IP ages out, following this correction, unless
1016	   additional action is taken to speed up recovery.
1017
1018	   This section lists possible additional corrective actions that could
1019	   be taken to achieve faster recovery to normal operation.
1020
1021	8.4.1.  Route Un-freezing Configuration
1022
1023	   Unfreezing the duplicate or frozen MAC or IP via a CLI can be used to
1024	   recover from duplicate and frozen state following corrective un-
1025	   provisioning of the duplicate MAC or IP.
1026
1027	   Unfreezing the frozen MAC or IP via a CLI at a PE should result in
1028	   that MAC or IP being advertised with a sequence number that is higher
1029	   than the sequence number advertised from the other location of that
1030	   MAC or IP.
1031
1032	   Two possible corrective un-provisioning scenarios exist:
1033
1034	   *  Scenario A: A duplicate MAC or IP may have been un-provisioned at
1035	      the location where it was NOT marked as duplicate and frozen.
1036
1037	   *  Scenario B: A duplicate MAC or IP may have been un-provisioned at
1038	      the location where it was marked as duplicate and frozen.
1039
1040	   Unfreezing the duplicate and frozen MAC or IP, following the above
1041	   corrective un-provisioning scenarios would result in recovery to
1042	   steady state as follows:
1043
1044	   *  Scenario A: If the duplicate MAC or IP was un-provisioned at the
1045	      location where it was NOT marked as duplicate, unfreezing the
1046	      route at the frozen location will result in the route being
1047	      advertised with a higher sequence number.  This would in-turn
1048	      result in automatic clearing of local route at the PE location,
1049	      where the host was un-provisioned via ARP/ND PROBE and DELETE
1050	      procedure specified earlier in section 6 and in [RFC7432].
1051
1052	   *  Scenario B: If the duplicate host is un-provisioned at the
1053	      location where it was marked as duplicate, unfreezing the route
1054	      will trigger an advertisement with a higher sequence number to the
1055	      other location.  This would in-turn trigger re-learning of local
1056	      route at the remote location, resulting in another advertisement
1057	      with a higher sequence number from the remote location.  Route at
1058	      the local location would now be cleared on receiving this remote
1059	      route advertisement, following the ARP/ND PROBE.
1060
1061	   Note that the probes referred to in the above scenarios are event
1062	   driven probes resulting from receiving a route with a higher sequence
1063	   number.  Periodic probes resulting from refresh timers may also occur
1064	   in addition as completely independent probes.
1065
1066	8.4.2.  Route Clearing Configuration
1067
1068	   In addition to the above, route clearing CLIs may also be used to
1069	   clear the local MAC or IP route, to be executed AFTER the duplicate
1070	   host is un-provisioned:
1071
1072	   *  clear MAC CLI: A clear MAC CLI can be used to clear a duplicate
1073	      MAC route, to recover from a duplicate MAC scenario.
1074
1075	   *  clear ARP/ND: A clear ARP/ND CLI may be used to clear a duplicate
1076	      IP route to recover from a duplicate IP scenario.
1077
1078	   Note that the route unfreeze CLI may still need to be run if the
1079	   route was un-provisioned and cleared from the non-duplicate / non-
1080	   frozen location.  Given that unfreezing of the route via the un-
1081	   freeze CLI would any ways result in auto-clearing of the route from
1082	   the "un- provisioned" location, as explained in the prior section,
1083	   need for a route clearing CLI for recovery from duplicate / frozen
1084	   state is truly optional.

[major]
* what is the 0-bit=0? please add a specific reference

[re-edit]
8. Duplicate Host Detection

Duplicate host detection scenarios across EVPN IRB can be classified as follows:

* Scenario A: Two hosts have the same MAC address (host IPs may or may not be duplicates).
* Scenario B: Two hosts have the same IP address but different MAC addresses.
* Scenario C: Two hosts have the same IP address, and the host MAC is not advertised.

Duplicate detection procedures for Scenarios B and C do not apply to "anycast IPv6" hosts advertised via NA messages with 0-bit=0, as per [RFC9161].

8.1. Scenario A

In cases where duplicate hosts share the same MAC address, the MAC is detected as duplicate using the duplicate MAC detection procedure described in [RFC7432]. Corresponding MAC-IP routes with the same MAC do not require separate duplicate detection and MUST inherit the duplicate property from the MAC route. If a MAC route is marked as duplicate, all associated MAC-IP routes MUST also be treated as duplicates. Duplicate detection procedures need only be applied to MAC routes.

8.2. Scenario B

Misconfigurations may lead to different MAC addresses being assigned the same IP address. This scenario is not detected by [RFC7432] duplicate MAC detection procedures and can result in incorrect routing of traffic destined for the IP address.

Such situations, when detected locally, are identified as a move scenario through the local MAC sequence number computation procedure described in section 6.1:

* If the IP is associated with a different remote MAC "Mz," the sequence number MUST be higher than the "Mz" sequence number.

This move results in a sequence number increment for the local MAC due to the remote MAC-IP route associated with a different MAC, counting as an "IP move" against the IP, independent of the MAC. The duplicate detection procedure described in [RFC7432] can then be applied to the IP entity independent of the MAC. Once an IP is detected as duplicate, the corresponding MAC-IP route should be treated as duplicate. Associated MAC routes and any other MAC-IP routes related to this MAC should not be affected.

8.2.1. Duplicate IP Detection Procedure for Scenario B

The duplicate IP detection procedure for this scenario is specified in [RFC9161]. An "IP move" is further clarified as follows:

* Upon learning a local MAC-IP route Mx-IPx, check for existing remote or local routes for IPx with a different MAC association (Mz-IPx). If found, count this as an "IP move" for IPx, independent of the MAC.

* Upon learning a remote MAC-IP route Mz-IPx, check for existing local routes for IPx with a different MAC association (Mx-IPx). If found, count this as an "IP move" for IPx, independent of the MAC.

A MAC-IP route SHOULD be treated as duplicate if either:

* The corresponding MAC route is marked as duplicate via the existing detection procedure.

* The corresponding IP is marked as duplicate via the extended procedure described above.

8.3. Scenario C

In a purely routed overlay scenario, as described in section 7, where only a host IP is advertised via EVPN RT-5 with a sequence number mobility attribute, duplicate MAC detection procedures specified in [RFC7432] can be applied intuitively to IP-only host routes for duplicate IP detection.

* Upon learning a local host IP route IPx, check for existing remote or local routes for IPx with a different ESI association. If found, count this as an "IP move" for IPx.

* Upon learning a remote host IP route IPx, check for existing local routes for IPx with a different ESI association. If found, count this as an "IP move" for IPx.

* Using configurable parameters "N" and "M," if "N" IP moves are detected within "M" seconds for IPx, IPx should be treated as duplicate.

8.4. Duplicate Host Recovery

Once a MAC or IP is marked as duplicate and frozen, corrective action must be taken to un-provision one of the duplicate MAC or IP addresses. Un-provisioning refers to corrective action taken on the host side. Following this correction, normal operation will not resume until the duplicate MAC or IP ages out unless additional action is taken to expedite recovery.

Possible additional corrective actions for faster recovery include:

8.4.1. Route Unfreezing Configuration

Unfreezing the duplicate or frozen MAC or IP via a CLI can be used to recover from the duplicate and frozen state following corrective un-provisioning of the duplicate MAC or IP. Unfreezing the MAC or IP should result in advertising it with a sequence number higher than that advertised from the other location.

Two scenarios exist:

* Scenario A: The duplicate MAC or IP is un-provisioned at the location where it was not marked as duplicate.

* Scenario B: The duplicate MAC or IP is un-provisioned at the location where it was marked as duplicate.

Unfreezing the duplicate and frozen MAC or IP will result in recovery to a steady state as follows:

* Scenario A: If the duplicate MAC or IP is un-provisioned at the non-duplicate location, unfreezing the route at the frozen location results in advertising with a higher sequence number, leading to automatic clearing of the local route at the un-provisioned location via ARP/ND PROBE and DELETE procedures.

* Scenario B: If the duplicate host is un-provisioned at the duplicate location, unfreezing the route triggers an advertisement with a higher sequence number to the other location, prompting re-learning and clearing of the local route at the original location upon receiving the remote route advertisement.

Probes referred to in these scenarios are event-driven probes resulting from receiving a route with a higher sequence number. Periodic probes resulting from refresh timers may also occur independently.

8.4.2. Route Clearing Configuration

In addition to the above, route clearing CLIs may be used to clear the local MAC or IP route after the duplicate host is un-provisioned:

* Clear MAC CLI: Used to clear a duplicate MAC route.

* Clear ARP/ND: Used to clear a duplicate IP route.

The route unfreeze CLI may still need to be executed if the route was un-provisioned and cleared from the non-duplicate location. Given that unfreezing the route via the CLI would result in auto-clearing from the un-provisioned location, as explained earlier, using a route clearing CLI for recovery from the duplicate state is optional.

Kind Regards,
Gunter Van de Velde
Routing Area Director