[bess] [Shepherding AD review] review of draft-ietf-bess-evpn-fast-df-recovery-08

["Gunter van de Velde (Nokia)" <[e-mail]>]

# Gunter Van de Velde, RTG AD, comments for draft-ietf-bess-evpn-fast-df-recovery-08

Hi All,

Please find here a shepherding AD review of draft-ietf-bess-evpn-fast-df-recovery-08

I'm sorry it took a bit of time to get started on this draft.

I've begun reviewing this document before we kick off the IETF Last Call process. Once we address these points, we can move forward with the document through the IESG chain.

A big thank you to Adrian Farrel for his RTG-DIR review on the -07 version, which helped improve the document to its -08 version and to Matthew Bocci for the Shepherds write-up (4 July 2022)

In my review, I've noted some final observations while going through the document. For better readability, I've suggested some paragraph edits.

One thing I noticed is that there's not much RFC 2119-based normative language used. Maybe the authors can take another look and add or update the RFC 2119 text where needed.

You can find my review notes below.

#GENERIC COMMENTS
#================

88	   Virtualization Overlay (NVO) and DC inte)rconnect (DCI) services, and

Typo with the ")"

100	   multihomed Ethernet Segment.  This DF election is achieved
101	   independent of the number of EVPN Instances (EVIs) associated with
102	   that Ethernet Segment and it is performed via simple signaling
103	   between the recovered node and each of the other nodes in the
104	   multihomed group.

I believe that the word 'simple' is reasonable subjective. It may be better to replace with a construct using 'straightforward'. Possible rewrite:

"This Designated Forwarder (DF) election is conducted independently of the number of EVPN Instances (EVIs) associated with the Ethernet Segment and is executed through straightforward signaling between the recovered node and each of the other nodes in the multihomed group.
"

105	   This document updates the state machine described in Section 2.1 of

Being more explicit in what is updated could be better.

"This document updates the DF Election Finite State Machine (FSM) described in Section 2.1 of"

131	   In EVPN technology, multiple Provider Edge (PE) devices have the
132	   ability to encap and decap data belonging to the same VLAN.  In

expand on encap and decap for better readability.

131	   In EVPN technology, multiple Provider Edge (PE) devices have the
132	   ability to encap and decap data belonging to the same VLAN.  In
133	   certain situations, this may cause L2 duplicates and even loops if
134	   there is a momentary overlap of forwarding roles between two or more
135	   PE devices, leading to broadcast storms.

possible readability rewrite:
"In EVPN technology, multiple Provider Edge (PE) devices possess the capability to encapsulate and decapsulate data associated with the same VLAN. Under certain conditions, this may result in Layer 2 duplicates and potential loops if there is a temporary overlap in forwarding roles among two or more PE devices, consequently leading to broadcast storms.
"
137	   EVPN [RFC7432] currently uses timer based synchronization among PE
138	   devices in a redundancy group that can result in duplications (and
139	   even loops) because of multiple DFs if the timer is too short or
140	   packets being dropped if the timer is too long.

RFC7432 is providing more a specification the using a timer. Hence a more explicit text blob to document this property: 

"EVPN [RFC7432] currently specifies timer-based synchronization among PE devices within a redundancy group. This approach can lead to duplications and potential loops due to multiple Designated Forwarders (DFs) if the timer interval is too short, or to packet drops if the timer interval is too long."

142	   Using split-horizon filtering (Section 8.3 of [RFC7432]) can prevent
143	   loops (but not duplicates).  However, if there are overlapping DFs in
144	   two different sites at the same time for the same VLAN, the site
145	   identifier will be different upon the packet re-entering the Ethernet
146	   Segment and hence the split-horizon check will fail, leading to L2
147	   loops.

Strange grammatical construct and usage of "()". Potential rewrite to correct this assuming i kept the issue described correct:

"Employing split-horizon filtering, as described in Section 8.3 of [RFC7432], can prevent loops but does not address duplicates. However, if there are overlapping Designated Forwarders (DFs) at two different sites simultaneously for the same VLAN, the site identifier will differ when the packet re-enters the Ethernet Segment. Consequently, the split-horizon check will fail, resulting in Layer 2 loops.
"

149	   The updated DF procedures in [RFC8584] use the well known Highest
150	   Random Weight (HRW) algorithm to avoid reshuffling of VLANs among PE
151	   devices in the redundancy group upon failure/recovery.  This reduces
152	   the impact to VLANs not assigned to the failed/recovered ports and
153	   eliminates loops or duplicates at failure/recovery events.

Is there a reference that can be used for the well known HRW algorithm? 
What about the following rewrite proposal for readability:

"The updated Designated Forwarder (DF) procedures outlined in [RFC8584] utilize the well-known Highest Random Weight (HRW) algorithm to prevent the reshuffling of VLANs among PE devices within the redundancy group during failure or recovery events. This approach minimizes the impact on VLANs not assigned to the failed or recovered ports and eliminates the occurrence of loops or duplicates during such events.
"

179	   a given VLAN is possible.  Duplication of DF roles may eventually
180	   lead to duplication of traffic as well as L2 loops.

in previous text the word 'overlap' was used while here the word Duplication of DF roles is used.

195	   *  Complicated handshake signamling mechanisms and state machines are
196	      avoided in favor of a simple uni-directional signaling approach.

s/Complicated/Complex/
s/signamling/signaling/

198	   *  The solution is backwards-compatible (see Section 4), by PEs
199	      simply discarding the unrecognized new BGP Extended Community.

I think that the "The solution" seems reasonable opaque description. Maybe we should explicit mention that this concerns the fast dr recovery solution. I only noted this here as the first occurrence, but the more explicit text can be used in multiple locations within the draft text.

What about:
"The fast df recovery solution maintains backwards compatibility (see Section 4) by ensuring that PEs discard any unrecognized new BGP Extended Community."

201	   *  Existing DF Election algorithms are supported.

s/are/remain/

232	   Upon receipt of that new BGP Extended Community, partner PEs can
233	   determine the service carving time of the newly insterted PE.  The
234	   notion of skew is introduced to eliminate any potential duplicate
235	   traffic or loops.  The receiving partner PEs add a skew (default =
236	   -10ms) to the Service Carving Time to enforce this.  The previously
237	   inserted PE(s) must carve first, followed shortly (skew) by the newly
238	   insterted PE.

I got thrown off-guard with the word skew as a non-native English speaker.
Maybe a small explanation would be helpful. What about the following:

"Upon receipt of the new BGP Extended Community, partner PEs can determine the service carving time of the newly inserted PE. To eliminate any potential for duplicate traffic or loops, the concept of skew-a small time delay added to the service carving process to ensure a controlled and orderly transition when multiple Provider Edge (PE) devices are involved-is introduced. The receiving partner PEs add a skew (default = -10ms) to the service carving time to enforce this mechanism. This ensures that the previously inserted PEs complete their carving process first, followed shortly thereafter (by the specified skew) by the newly inserted PE.
"

240	   To summarize, all peering PEs carve almost simultaneously at the time
241	   announced by the newly added/recovered PE.  The newly inserted PE
242	   initiates the SCT, and carves immediately on its local timer expiry.
243	   The previously inserted PE(s) receiving Ethernet Segment route (RT-4)
244	   with a SCT BGP extended community, carve shortly before Service
245	   Carving Time.

This text provides me some confusion. The term "to carve" generally means to cut or shape something from a larger piece, often with precision and care. Hence i was a bit surprised to see this used here. 

May I assume that in the context of these network operations and specifically within EVPN (Ethernet VPN) and MPLS (Multiprotocol Label Switching) environments, "to carve" typically refers to the process of determining and establishing roles or responsibilities for forwarding traffic among Provider Edge (PE) devices? 

If yes, maybe such text blob should be explicit mentioned somewhere in the draft?

266	   [RFC5905].  As the current NTP era value is not exchanged, a local
267	   clock which is "synchronized" but to the wrong era is outside of the
268	   scope of this document.

What is era value?

257	                        1                   2                   3
258	    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
259	   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
260	   | Type = 0x06   | Sub-Type(0x0F)|      Timestamp Seconds        ~
261	   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
262	   ~  Timestamp Seconds            | Timestamp Fractional Seconds  |
263	   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

a figure number/caption is missing.

269	   The 64-bit timestamp of NTP consists of a 32-bit part for seconds and
270	   a 32-bit part for fractional second:

There seems to be a 32bit/64bit and 128bt timestamp according:
https://datatracker.ietf.org/doc/html/rfc5905#section-6
Should description not align with all of these?

274	   *  Timestamp Fractional Seconds: the high order 16 bits of the NTP
275	      fractional seconds are encoded in this field.  The use of a 16-bit
276	      fractional seconds yields adequate precision of 15 microseconds
277	      (2^-16 s).

I assume that the lower order 16 bits are assumed to be '0'? Maybe that should be explicit called out?

296	   This capability is used in conjunction with the agreed upon DF Type
297	   (DF Election Type).  For example if all the PEs in the Ethernet
298	   Segment indicate having Time Synchronization capability and are
299	   requesting the DF type to be HRW, then the HRW algorithm is used in
300	   conjunction with this capability.

readability rewrite:
"This capability is utilized in conjunction with the agreed-upon Designated Forwarder (DF) Type (DF Election Type). For instance, if all the PE devices in the Ethernet Segment indicate possessing Time Synchronization capability and request the DF Type to be Highest Random Weight (HRW), then the HRW algorithm is employed in conjunction with this capability.
"

Note, what happens if one of the involved PEs do not support Time synchronisation capability?

309	   The peering PE's FSM in DF_DONE which receives a RECV_ES transitions
310	   to DF_CALC.  Because of the SCT carried in the Ethernet-Segment
311	   update, the output of the DF_CALC and transition back into DF_DONE
312	   are delayed, as are accompanying forwarding updates to DF/NDF state.

This processes not so easy. I assume that all these are states of the FSM?
Would the following be a correct rewrite for readability?

"Upon receiving a RECV_ES message, the peering PE's Finite State Machine (FSM) transitions from the DF_DONE (indicating the DF election process was complete) state to the DF_CALC (indicating that a new DF calculation is needed) state . Due to the Service Carving Time (SCT) included in the Ethernet-Segment update, the completion of the DF_CALC state and the subsequent transition back to the DF_DONE state are delayed. This delay ensures proper synchronization and prevents conflicts. Consequently, the accompanying forwarding updates to the Designated Forwarder (DF) and Non-Designated Forwarder (NDF) states are also deferred.
"

314	   The corresponding actions when transitions are performed or states
315	   are entered/exited is modified as follows:
316
317	   9.  DF_CALC on CALCULATED: Mark the election result for the VLAN or
318	       Bundle.
319
320	       9.1  Where SCT timestamp is present on the RECV_ES event of
321	            Action 11, wait until the time indicated by the SCT before
322	            continuing to 9.2.
323
324	       9.2  Assume a DF/NDF for the local PE for the VLAN or VLAN
325	            Bundle, and transition to DF_DONE.

What about the following procedure text blob description for clarity:

"
The corresponding actions when transitions are performed or states are entered/exited are modified as follows:

9. DF_CALC on CALCULATED: Mark the election result for the VLAN or VLAN Bundle.

9.1. If an SCT timestamp is present during the RECV_ES event of Action 11, wait until the time indicated by the SCT before proceeding to step 9.2.

9.2. Assume the role of DF or NDF for the local PE concerning the VLAN or VLAN Bundle, and transition to the DF_DONE state.

This revised approach ensures proper timing and synchronization in the DF election process, avoiding conflicts and ensuring accurate forwarding updates.
"

329	   Let's take Figure 1 as an example where initially PE2 had failed and
330	   PE1 had taken over.  This example shows the problem with the
331	   DF-Election mechanism in Section 8.5 of [RFC7432], using the value of
332	   the timer configured for all PEs on the Ethernet Segment.

To make the text more proposed standard style, what about this textblob for readability:

"Consider Figure 1 as an example, where initially PE2 has failed and PE1 has taken over. This scenario illustrates the problem with the DF-Election mechanism described in Section 8.5 of [RFC7432], specifically in the context of the timer value configured for all PEs on the Ethernet Segment.
"

334	   Based on Section 8.5 of [RFC7432] and using the default 3 second
335	   timer in step 2:
337	   1.  Initial state: PE1 is in steady-state, PE2 is recovering
339	   2.  PE2 recovers at (absolute) time t=99
341	   3.  PE2 advertises RT-4 (sent at t=100) to partner PE1
343	   4.  PE2 starts a 3 second timer to allow the reception of RT-4 from
344	       other PE nodes
346	   5.  PE1 carves immediately on RT-4 reception, i.e. t=100 + minimal
347	       BGP propagation delay
349	   6.  PE2 carves at time t=103
350
351	   [RFC7432] aims of favouring traffic being dropped over duplicate
352	   traffic.  With the above procedure, traffic drops will occur as part
353	   of each PE recovery sequence since PE1 has transitioned some VLANs to
354	   Non-Designated-Forwarder (NDF) immediately upon reception.
355	   The timer value (default = 3 seconds) has a direct effect on the
356	   duration of the packets being dropped.  A shorter (especially zero)
357	   timer may, however, result in duplicate traffic or traffic loops.

What about:

"Procedure Based on Section 8.5 of [RFC7432] with Default 3-Second Timer:
1. Initial State: PE1 is in a steady state, and PE2 is recovering.
2. Recovery: PE2 recovers at an absolute time of t=99.
3. Advertisement: PE2 advertises RT-4, sent at t=100, to partner PE1.
4. Timer Start: PE2 starts a 3-second timer to allow the reception of RT-4 from other PE nodes.
5. Immediate Carving: PE1 carves immediately upon RT-4 reception, i.e., t=100 plus minimal BGP propagation delay.
6. Delayed Carving: PE2 carves at time t=103.

[RFC7432] favors traffic drops over duplicate traffic. With the above procedure, traffic drops will occur as part of each PE recovery sequence since PE1 transitions some VLANs to Non-Designated Forwarder (NDF) immediately upon RT-4 reception. The timer value (default = 3 seconds) directly affects the duration of the packet drops. A shorter (or zero) timer may result in duplicate traffic or traffic loops.
"

359	   Based on the Service Carving Time (SCT) approach:
361	   1.  Initial state: PE1 is in steady-state, PE2 is recovering
363	   2.  PE2 recovers at (absolute) time t=99
365	   3.  PE2 advertises RT-4 (sent at t=100) with target SCT value t=103
366	       to partner PE1
368	   4.  PE2 starts a 3 second timer to allow the reception of RT-4 from
369	       other PE nodes
371	   5.  PE1 starts service carving timer, with remaining time until t=103
373	   6.  Both PE1 and PE2 carve at (absolute) time t=103
374	   In fact, PE1 should carve slightly before PE2 (skew) to maintain the
375	   preference of minimal loss over duplicate traffic.  The previously
376	   inserted PE2 that is recovering performs both transitions DF to NDF
377	   and NDF to DF per VLANs at the timer's expiry.  Since the goal is to
378	   prevent duplicates, the original PE1, which received the SCT will
379	   apply:
381	   *  DF to NDF transition at t=SCT minus skew, where both PEs are NDF
382	      for 'skew' amount of time
384	   *  NDF to DF transition at t=SCT
385
386	   It is this split-behaviour which ensures a good transition of DF role
387	   with contained amount of loss.
388
389	   Using SCT approach, the negative effect of the timer to allow the
390	   reception of RT-4 from other PE nodes is mitigated.  Furthermore, the
391	   BGP Ethernet Segment route (RT-4) transmission delay (from PE2 to
392	   PE1) becomes a non-issue.  The use of SCT approach remedies the
393	   problem associated with this timer: the 3 second timer window is
394	   shortened to the order of milliseconds.

What about the following textblobs for readability:

"Procedure Based on the Service Carving Time (SCT) Approach:
1. Initial State: PE1 is in a steady state, and PE2 is recovering.
2. Recovery: PE2 recovers at an absolute time of t=99.
3. Advertisement: PE2 advertises RT-4, sent at t=100, with a target SCT value of t=103 to partner PE1.
4. Timer Start: PE2 starts a 3-second timer to allow the reception of RT-4 from other PE nodes.
5. Service Carving Timer: PE1 starts the service carving timer, with the remaining time until t=103.
6. Simultaneous Carving: Both PE1 and PE2 carve at an absolute time of t=103.

To maintain the preference for minimal loss over duplicate traffic, PE1 should carve slightly before PE2 (with skew). The recovering PE2 performs both DF to NDF and NDF to DF transitions per VLAN at the timer's expiry. The original PE1, which received the SCT, applies the following:

* DF to NDF Transition: At t=SCT minus skew, where both PEs are NDF for the skew duration.
* NDF to DF Transition: At t=SCT.

This split-behavior ensures a smooth DF role transition with minimal loss.

Using the SCT approach, the negative effect of the timer to allow the reception of RT-4 from other PE nodes is mitigated. Furthermore, the BGP Ethernet Segment route (RT-4) transmission delay (from PE2 to PE1) becomes a non-issue. The SCT approach shortens the 3-second timer window to the order of milliseconds, addressing the associated problems.
"

396	3.1.  Concurrent Recoveries

This section seems to be missing RFC2119 language on how nodes need to behave with respect the procedures outlined in this document.

402	   Election.  A similar situation arises in staggered recovering PEs,
403	   when a second PE recovers at rougly a first PE's advertised SCT
404	   expiry, and with its own new SCT-2 outside of the initial SCT window.

The word staggered is oddly used. What about the following:

"A similar situation arises in sequentially recovering PEs, when a second PE recovers approximately at the time of the first PE's advertised SCT expiry, and with its own new SCT-2 outside of the initial SCT window."

406	   In the case of multiple outstanding DF elections, one requested by
407	   each of the recovering PEs, the SCTs must simply be time-ordered and
408	   all PEs execute only a single DF Election at the service carving time
409	   corresponding to the largest received timestamp value.  The DF
410	   Election will involve all the active PEs in a single DF Election
411	   update.

To add to a similar edited writing style:
"In the case of multiple concurrent DF elections, each initiated by one of the recovering PEs, the SCTs must be ordered chronologically. All PEs shall execute only a single DF Election at the service carving time corresponding to the latest received timestamp value. This DF Election will involve all active PEs in a unified DF Election update.
"

However, it may require some formal RFC2119 language to make sure that implementations behave according this procedure

413	   Example:
415	   1.  Initial state: PE1 is in steady-state, all services elected at
416	       PE1.
418	   2.  PE2 recovers at time t=100, advertises RT-4 with target SCT value
419	       t=103 to partners (PE1)
421	   3.  PE2 starts a 3 second timer to allow the reception of RT-4 from
422	       other PE nodes
424	   4.  PE1 starts service carving timer, with remaining time until t=103
426	   5.  PE3 recovers at time t=102, advertises RT-4 with target SCT value
427	       t=105 to partners (PE1, PE2)
429	   6.  PE3 starts a 3 second timer to allow the reception of RT-4 from
430	       other PE nodes
432	   7.  PE2 cancels the running timer, starts service carving timer with
433	       remaining time until t=105
435	   8.  PE1 updates service carving timer, with remaining time until
436	       t=105
438	   9.  PE1, PE2 and PE3 carve at (absolute) time t=105

Example:
1. Initial State: PE1 is in a steady state, with all services elected at PE1.
2. Recovery of PE2: PE2 recovers at time t=100 and advertises RT-4 with a target SCT value of t=103 to its partners (PE1).
3. Timer Initiation by PE2: PE2 starts a 3-second timer to allow the reception of RT-4 from other PE nodes.
4. Timer Initiation by PE1: PE1 starts the service carving timer, with the remaining time until t=103.
5. Recovery of PE3: PE3 recovers at time t=102 and advertises RT-4 with a target SCT value of t=105 to its partners (PE1, PE2).
6. Timer Initiation by PE3: PE3 starts a 3-second timer to allow the reception of RT-4 from other PE nodes.
7. Timer Update by PE2: PE2 cancels the running timer and starts the service carving timer with the remaining time until t=105.
8. Timer Update by PE1: PE1 updates its service carving timer, with the remaining time until t=105.
9. Service Carving: PE1, PE2, and PE3 perform service carving at the absolute time of t=105.

446	4.  Backwards Compatibility
447
448	   Per redundancy group, for the DF election procedures to be globally
449	   convergent and unanimous, it is necessary that all the participating
450	   PEs agree on the DF Election algorithm to be used.  It is, however,
451	   possible that some PEs continue to use the existing modulo-based DF
452	   election and do not rely on the new SCT BGP extended community.  PEs
453	   running a baseline DF election mechanism will simply discard the new
454	   SCT BGP extended community as unrecognized.
455
456	   A PE can indicate its willingness to support clock-synched carving by
457	   signaling the new 'T' DF Election Capability as well as including the
458	   new Service Carving Time BGP extended community along with the
459	   Ethernet Segment Route (Type-4).  In the case where one or more PEs
460	   attached to the Ethernet Segment do not signal T=1, all PEs in the
461	   Ethernet Segment SHALL revert back to the [RFC7432] timer approach.
462	   This is especially important in the context of the VLAN shuffling
463	   with more than 2 PEs.

I am not sure what the modulo-based df is? is that the rfc7432 procedure? It was the first time that this was mentioned in this draft i believe.

what about following rewrite proposal for readability, but please add reference for the modulo-based election:

"For the DF election procedures to achieve global convergence and unanimity within a redundancy group, it is essential that all participating PEs agree on the DF election algorithm to be employed. However, it is possible that some PEs may continue to use the existing modulo-based DF election algorithm and not utilize the new Service Carving Time (SCT) BGP extended community. PEs that operate using the baseline DF election mechanism will simply discard the new SCT BGP extended community as unrecognized.

A PE can indicate its willingness to support clock-synchronized carving by signaling the new 'T' DF Election Capability and including the new SCT BGP extended community along with the Ethernet Segment Route (Type-4). If one or more PEs attached to the Ethernet Segment do not signal T=1, then all PEs in the Ethernet Segment SHALL revert to the timer-based approach as specified in [RFC7432]. This reversion is particularly crucial in preventing VLAN shuffling when more than two PEs are involved"

465	5.  Security Considerations

The conditions for when the SCT is far away in the future, it was not entirely clear or spelled out what an implementation should do. Maybe make it more explicite in the textual decscription as a normative reference using RFC2119 language