Re: [Idr] WG LC for draft-ietf-idr-bgp-car-02 (7/10/2023 to 7/24/2023)

["Dhananjaya Rao (dhrao)" <[e-mail]>]

Hi Nats,

Thank you for the review. Please see inline (DR#)

From: Idr <[e-mail]> on behalf of Natrajan Venkataraman <natv=[e-mail]>
Date: Thursday, November 9, 2023 at 7:26 AM
To: Susan Hares <[e-mail]>, [e-mail] <[e-mail]>
Subject: Re: [Idr] WG LC for draft-ietf-idr-bgp-car-02 (7/10/2023 to 7/24/2023)
Hi Susan and WG members,

The following review is for

- BGP-CAR-03 (draft-ietf-idr-bgp-car-03)

with respect to the intent-driven service mapping problem
and how it addresses provider network deployment scenarios
that are seen today, while mainly focusing on the issues
raised in IDR GitHub by the CT team.

  - https://github.com/ietf-wg-idr/draft-ietf-idr-bgp-car/issues

    1) F3-CAR-Issue-1: BGP-CAR Appendix A.7 – Anycast EP Scenario:
    2) F3-CAR-Issue-2: BGP-CAR – Consensus on the need for resolution-schemes
    3) F3-CAR-Issue-3: CAR-Q3 - Handling [of] LCM and Color-EC
    4) F3-CAR-Q4: Mis-Routing in Non-agreeing color-domains for Anycast EPs
    5) F3-CAR-Q5: Update Packing Observations

Having spent considerable amount of time for this review, I hope it
helps operators/vendors to,

- Get closer to this problem space
- Understand how CAR achieves different aspects of service mapping
- Get implementation insights for CAR based service mapping
- How different deployment scenarios are handled by this draft

Verdict: Has Issues
===================


Setting up Key Aspects:
=======================

Quoting from RFC 1925: Twelve Networking Truths:

   (4)  Some things in life can never be fully appreciated nor
        understood unless experienced firsthand. Some things in
        networking can never be fully understood by someone who neither
        builds commercial networking equipment nor runs an operational
        network.

Before we begin,
Intent-driven service mapping using BGP is a substantial effort.
It applies to "all" BGP Service Families whose nexthop is reachable
via transport tunnels. These tunnels belong to various forwarding
architectures such as MPLS (RSVP-TE, SRTE), IPv6 (SRv6), IPinIP or
MPLSoverUDP. It introduces new constructs that allow for an operator
to be able to specify the service intent. Therefore, this is a substantial
upgrade for existing provider network deployment scenarios such as
MPLS-VPN Options, CsC, Multi-Homed CE and Anycast scenarios that are
prevalent today. Therefore, this review will evaluate BGP-CAR based on
certain key aspects and operational criteria.

Intent-Driven Service Mapping Key Aspects:
------------------------------------------

To better understand the nature of issues raised, it is important
to consider the basic intent-driven service mapping aspects.

A) Classification & Grouping:

    Construct to group underlay tunnels with sufficiently similar TE
    characteristics

      - CAR: Classification: Color
             Grouping: None (Not clear from the draft if any construct is used)

DR# Color is the construct for an intent, already well-defined as per RFC 9256 - described in the introduction and main sections

B) Resolution & Steering:

    Construct for overlay routes with a Mapping Community e.g. Color-EC, to
    resolve next hop reachability to the underlay path with the same
    "effective color" arrived from the following

      - CAR: Resolution for CAR routes &
             SR policy based steering/automatic-steering for Service routes
             > Color-EC on Service Route (Mapping Community)
             > CAR Type-1: IP:Color as key in NRLI
             > CAR Type-2: Colored IP Prefix
             > Color-EC on CAR route for Anycast
             > LCM-EC for non-agreeing color-domains

    and download service routes to FIB when its path is reachable

C) BGP Extensions: (AFI/SAFI Encoding/Decoding)

    A BGP family to extend the above constructs to adjacent domains (AS/IGP)

      - CAR NLRIs: (New NLRI formats for CAR and VPN-CAR)
        > Type-1: Color as key in NRLI
        > Type-2: IP Prefix as Color
        > Color-EC attr (for Anycast)
        > LCM-EC attr (for non-agreeing color domains)
        > IP prefix for CAR types and RD:IP prefix for VPN-CAR types

D) Path Availability & Selection:

    This is the final and most important aspect which has two important
    steps for selecting transport routes for nexthop reachability:

    i) Avoid path hiding -
        - CAR route: Type-1: Unique "IP:Color" key in NLRI
                     Type-2: Unique "Color IP Prefix" per color-locator

    ii) Path selection and Resolution should use the same "effective color"
        - CAR route: Type-1:
                     Path Selection key Prefix: IP and Type-1-NLRI-Color
                     Resolution key Prefix: IP and "Effective Color"
                                            (Type-1-NLRI-Color or
                                             Color-EC on CAR route or
                                             LCM-EC on CAR route)
                     Type-2:
                     Path Selection key Prefix: IP color prefix
                     Resolution key Prefix: Not needed as it is routed


Review:
=======

Switching back to the open issues and scenarios, this review will focus on
evaluating CAR on the above set of aspects and criteria.


MAJOR: "Where is my color?"

- From CAR-00, as a draft that addresses this "new" problem space of using
  "Color" to achieve "Intent driven service mapping", CAR introduces the
  "Where is my color?" ambiguity in procedures for any node receiving and
  processing a CAR route.

DR# The draft clearly specifies where color is derived from and for what purposes. Specific comments are addressed below.


  These are arrived from the following procedures:

  PROCEDURE I: CAR Route validation
  ---------------------------------

    Quoting from section 2.4: BGP CAR Route Validation

      "A BGP CAR path (E, C) from N with encapsulation T is valid if color-
       aware path (N, C) exists with encapsulation T available in dataplane."

      " A local policy may customize the validation process:

       *  the color constraint in the first check may be relaxed: instead N
          is reachable via alternate color(s) or in the default routing
          table"

          ...
      "A path that is not valid MUST NOT be considered for BGP best path
       selection."

  PROCEDURE II: Path Selection and Availibility
  ---------------------------------------------

    Quoting from section 2.7: Path Availibility

      "The (E, C) route inherently provides availability of redundant paths
       at every hop, identical to BGP-LU or BGP IP."

  PROCEDURE III: CAR Route resolution
  -----------------------------------

    Quoting from section 2.5: BGP CAR Route Resolution

      "For a CAR route, Color-EC color takes precedence over route NLRI color."

      "A BGP CAR route may recursively resolve over a BGP route carrying
       Tunnel Encapsulation attribute. Procedures of section 8 of [RFC9012]
       apply in presence of BGP Color EC in the CAR route. They are
       extended to use LCM EC and Color in CAR route NLRI as per above and
       Section 2.9.4 in absence of BGP Color EC."

    Quoting from section 2.9.4: LCM-EC

      "When a CAR route crosses the originator's color domain boundary, LCM-
       EC is added."

      "An implementation SHOULD NOT send more than one instance of the LCM-
       EC.  However, if more than one instance is received, an
       implementation MUST disregard all instances other than the one with
       the numerically highest value"

      "If present, LCM-EC is the effective intent of a BGP CAR route."

      "LCM-EC Color is used instead of the Color in CAR route NLRI for
       procedures described in earlier sections such as route validation,
       resolution, AIGP calculation and steering."

    Quoting from section 2.10: LCM and BGP Color Extended Community usage

      "Default order of processing for resolution in presence of LCM-EC is
      local policy, then BGP Color-EC color, and finally LCM-EC color"

  This CAR route will have one or more of the following depending on the
  NLRI type and the Scenario being addressed.

  -----------------------------------------------------------------------
    | CAR color variable         | Scenario
  -----------------------------------------------------------------------
  A.| Color-EC color             | Anycast, Single Domain N:M color
  B.| LCM-EC color               | Multiple color domains
  C.| CAR-NLRI-Type-1 'color'    | Default, used for path availability and
    |                            | selection additonally
  D.| CAR-NLRI-Type-2 'IP Prefix'| Defined in Section 2.9.3, 8, 9 and 10.2
  -------------------------------------------------------------------------

  Quoting from RFC 1925: Twelve Networking Truths:

   (3)  With sufficient thrust, pigs fly just fine. However, this is
        not necessarily a good idea. It is hard to be sure where they
        are going to land, and it could be dangerous sitting under them
        as they fly overhead.


  Issue-1.1: As mentioned in key aspects, the path selection key and the
             resolution key can be different depending on the scenario.
             This will cause misrouting issues such as the ones mentioned
             in F3-CAR-Q4 below. Specific text needs to be added to address
             this scenario.

DR# There is no practical misrouting issue as has been extensively discussed during the WG adoption call.
DR# See the summary of a substantive discussion between Bruno and Jeff.

https://mailarchive.ietf.org/arch/msg/idr/ZKcFdSSn02hFTqe6p4pNOgNjDYY/

DR# As discussed there and recommended by Jeff, we’ve added relevant text in the Operational / Manageability considerations
section 11.

  Issue-1.2: CAR-NLRI-Type-1 'color' is overloaded for path selection,
             path availability as well as intent. While this solves path
             hiding for Unicast EPs, there is path hiding @ ABR/ASBRs
             in anycast and multihomed EP scenarios, and as a result,
             all paths are not visible to the ingress domain and thus
             end-to-end, as both Prefix and Color are same in all paths.
             Therefore, overloading CAR-NLRI-Type-1 'color' as both
             the distinguisher (RD) as well as "Intent" needs
             to be explicitly called out for each scenario.

DR# “Anycast” requires the network to route to the closest endpoint, and this is naturally and inherently supported with an (E,C) model
Reference Appendix A.7 Example 1.

DR# For a specific intent requirement that the paths to specific egress domain/nodes be visible to an ingress PE, assigning a specific color for the egress domain/node is aligned with the intent, not overloading. This would be Appendix A.7 example 2. This was also discussed and noted back during Jeff’s initial comparison.

             Moreover,
             using EBGP add-path at ASBR to get around this will use
             AddPath-ID as the distinguisher which is again, not
             well deployed and limits the distinguisher to the scope
             of the EBGP session and not end-to-end (e.g. RD)

DR# CAR doesn’t mandate any use of Addpath-ID more than what CT does. Let’s please stop attributing these gratuitous claims.
https://mailarchive.ietf.org/arch/msg/idr/gQ2QX8RNzu7jQTJ0mawobNtledI/


  Issue 1.3: Migration from unicast to anycast is not seamless for E2E
             path availability use-cases. This can happen when a CE is
             multihomed to another PE in the egress domain. Operators
             should expect path hiding and will have to use A.7 workaround
             to have an additional variable, Color-EC which is attached
             to the CAR route, and is preferred over CAR-NLRI-Type-1
             'color' as the effective intent even in single color domains.
             This is further discussed in F3-CAR-Q1 below.

DR# To clarify, the Color-EC indicates the intent/color to be used for next-hop resolution, following the well-established semantics in RFC9256.
It provides flexibility to automate the resolution when the color for resolving the next-hop is different than that of the route, as in this case.


  Issue 1.4: Colorless Transport Tunnels with varied intents like RSVP-TE,
             MPLSoUDP and IPinIP are quite prevalent in existing brownfield
             and customers do foresee these deployments continuing into the
             next decade as well. However, there still needs to be construct
             to achive the aspect of classification and grouping in such
             deployments. How can local policy achieve this in the absence
             of 'transport color' and 'TEA'?

DR# Color is indeed the construct for resolution, augmented by local policy to map as needed to traditional, color-unaware mechanisms.
DR# Please see Section 2.5.


  F3-CAR-Q1: Status = Unresolved
  ------------------------------

    BGP-CAR Appendix A.7 – Anycast EP Scenario:
    https://mailarchive.ietf.org/arch/msg/idr/nAj25sX0x_lp09VEqUDSCxmDR_w/


    Note: This is not a solution but just a work around. While bug fixes in
          code are common, it is painful to see bug fixes and work arounds
          in new and evolutionary technology drafts.

    F3-CAR-Issue-1.1: To work around Issue-1.2 and Issue-1.3, each anycast
                      end-point is configured with a different CAR-NLRI-Type-1
                      'Color' to avoid path hiding and attaching the same
                      Color-EC to all EPs additionally. Does this sound
                      familiar? Of course, Prefix remains the same,
                      CAR-NLRI-Type-1 'Color' now becomes the RD and Color-EC
                      becomes the intent. In CAR, this comes with a lot of
                      local policy overhead @ PE/ASBRs to STOP automatic
                      resolution using CAR-NLRI-Type 'color' as intent and
                      replace it with Color-EC as "intent". This is the reason
                      for the first two quotes mentioned in procedure (III.)
                      and quotes in procedure (I.). This issue still exists
                      and the draft needs to call out policy overhead required
                      in PE/ASBRs.

DR# Color in the NLRI represents the route intent. The use of Color-EC provides an -automated- mechanism to resolve the next-hop and takes precedence over route color as the text you’re quoting clearly states.

    F3-CAR-Issue-1.2: Considering the second quote in procedure III. and
                      an Administrative domain with ColorSRTE based transport
                      tunnels, a CAR route can resolve over a Color SRTE route
                      with TEA. This means that for N anycast EPs, there are N
                      colors assigned and hence N color SRTE paths per AS domain
                      for each Color-EC. This multiplies the Color SRTE
                      forwarding state by a factor of N. This needs to be called
                      out in the draft.

DR# All the N anycast routes can resolve via the same SR-TE color, specified in the color-EC. This illustrates the benefits of the decoupling and indirection between route and next-hop colors.

    F3-CAR-Issue-1.3: Quoting from section A.7

                      "Both E2 (in egress domain 2) and E3 (in egress domain 3)
                      advertise Anycast (shared) IP (IP1, C1) with same label L1"

                      Labels are dynamic. If the same label has to be sent on
                      both, there needs some static configuration to coordinate
                      this. This should translate to a SHOULD clause in the
                      procedures. If an implementation is unable to do static
                      labels, then recommendations need to be provided to
                      address the same.

DR# Not sure what is specific to CAR SAFI here. Static MPLS labels / SR prefix-SIDs / anycast SIDs are all existing well-known features.

    F3-CAR-Issue-1.4: How does ECMP and protection work here? Based on
                      procedure (I.), a local policy can be used to discard
                      CAR-NLRI-Type-1 'Color' and use alternate colors, say
                      in Color-EC to resolve the CAR route. However, the
                      question that still remains is how the route needs
                      to be maintained in the RIB? Should the route be
                      installed as IP:C1, IP:C2? Since these prefixes are
                      unique, path selection cannot act on them. This is
                      another example of C1 and C2 used as "distinguisher"
                      rather than "intent" and hence the ambiguity due
                      to overloading. If they are installed as just IP,
                      which RIB they are installed to? This needs to be
                      explained in the draft.

DR# Section 2.7 describes the key is (E,C), so multipathing is automatic, does not require additional “import” as in CT.
DR# Appendix A.7 provides examples of two choices. Not sure what is ambiguous.
DR# Choice of color and control for specific intent use-cases is in the hands of the operator.


    F3-CAR-Issue-1.5: Section B.2 - N:M distribution
                      In Figure 12: N:M illustration, attaches atleast two
                      Color-ECs on the CAR route while overloading the
                      CAR-NLRI-Type-1 'color' as both distinguisher as well
                      as intent for unicast EPs. There are no procedures
                      as to which Color-EC should take precedence in which
                      AS/ABR domain. This needs to be clarified. It should
                      also note that there needs to be one additional Color-EC
                      on the CAR route for each M (M1, M2, ...).

DR# The example clearly illustrates how the entire e2e resolution can be automated by leveraging the power of hierarchy and the standard and widely implemented/deployed Color-EC mechanism from RFC 9256.
DR# In this example, the color assignments for domains across the network are intentionally and by design disjoint, hence the resolution will always occur deterministically regardless of the order.

    F3-CAR-Issue-1.6: Section B.2 - N:M distribution and Anycast
                      In Figure 12: N:M illustration, now with Anycast in play
                      and as per F3-CAR-Issue-1.1, there will be atleast "three"
                      Color-ECs, where Color-EC-Anycast represents Anycast plus
                      the "two" Color-ECs for the N:M distribution. As per A.7,
                      CAR-NLRI-Type-1 'color' now becomes the distinguisher and
                      Color-EC-Anycast becomes the intent. This is a "critical"
                      issue because there is no clear precedence in picking
                      the Color-EC assigned for Anycast. This needs to be
                      clarified as part of the draft.

DR# The Color-EC assignments need not change if the route is for an Anycast or Egress Domain Prefix. So there is no issue above.


  F3-CAR-Q2: Status = Unresolved
  ------------------------------

    BGP-CAR – Consensus on the need for resolution-schemes
    https://mailarchive.ietf.org/arch/msg/idr/g6ZCJYzWwgRsilWlZY74MTv9Fqk/

    F3-CAR-Issue-2.1: Based on the above issue list, especially Issue-1.1,
                      Issue-1.4, F3-CAR-Issue-1.1, 1.4, 1.5 and 1.6, it is
                      indeed important for CAR to have a clear scheme for
                      resolution that handles all the of these issues. The
                      nature of this scheme needs to be explained as part of
                      this draft specifically, that vendors can use to
                      incorporate during implementation. Offloading this to
                      a local policy blob is prone to make implementations
                      incompatible due to configuration ambiguity.

DR# None of the above comments raised any specific issue that were not discussed previously or has not been addressed in the draft. Hope my responses have provided clarity. If some point is not clear, please let me know.

  F3-CAR-Q3: Status = Unresolved
  ------------------------------
    CAR-Q3 - Handling [of] LCM and Color Extended Communities
    https://mailarchive.ietf.org/arch/msg/idr/w5ROKVQPtVcI_BTBXfnKpKB4h4k/

    F3-CAR-Issue-3.1: Considering quotes from section 2.10 in Procedure (III.),
                      F3-CAR-Issue-1.1 to 1.6 apply. Additionally, LCM-EC needs
                      to be factored into F3-CAR-Issue-2.1.

    F3-CAR-Issue-3.2: Considering the following quotes from section 2.5,
                      2.9.4 and 2.10 in Procedure (III.) respectively,

                      "For a CAR route, Color-EC color takes precedence
                      over route NLRI color."

                      "When a CAR route crosses the originator's color
                      domain boundary, LCM-EC is added."

                      "If present, LCM-EC is the effective intent of
                      a BGP CAR route."

                      "Default order of processing for resolution in
                      presence of LCM-EC is local policy, then BGP Color-EC
                      color, and finally LCM-EC color"

                      These normative text contradict each other. If Color-EC
                      takes precedence over CAR-NLRI-Type-1 'color', then
                      Color-EC becomes the "intent" and CAR-NLRI-Type-1 'color'
                      becomes purely the distinguisher and not overloaded with
                      intent. In such cases, it is not very clear whether Color-EC
                      value should be the effective intent instead of
                      NLRI color and which one should be copied into LCM-EC since,
                      CAR-NLRI-Type-1 'color' is just a distinguisher.
                      This needs to be clarified as part of draft text.

DR# The text clearly states the above order of precedence is for CAR route resolution. Color-EC when present represents the intent/color used for resolving the BGP CAR next-hop, following semantics already established in RFC 9256. If not present, resolution uses the route color (whether its’ in NLRI or LCM). Let’s try to avoid cherry picking text out of context and mixing them.


  F3-CAR-Q4: Status = Unresolved
  ------------------------------

    BGP-CAR – Mis-Routing in Non-agreeing color-domains for Anycast EPs
    https://mailarchive.ietf.org/arch/msg/idr/OOZOBSyjdAYBar8NxvOqo6-5fAc/

    F3-CAR-Issue-4.1: Quoting from section 2.9.4

                      "If two BGP paths for a route have different LCM values,
                      it is considered an error and the route is not
                      considered for bestpath selection."

                      The above text still does not address the mis-routing
                      problems that might arise in Anycast with Non-Agreeing
                      color domains as indicated in the above issue link.

                      However, this seems to have been "worked around" using
                      Color-EC as the intent, CAR-NLRI-Type-1 'color' now
                      acting solely as a distinguisher and hence its derived
                      LCM-CE also acts as a distinguisher in the receiving
                      non-agreeing domain. The local policy operations for
                      this scenario needs to be clearly specified to avoid
                      ambiguity in the draft. It is also important to note
                      that a portion of colors from "intent namespace" is
                      being used purely as distinguishers and others being
                      overloaded with intent as well. This needs to be
                      clarified in the draft as to how the intent namespace(s)
                      are managed so that local policy schemes can derive how
                      this namespace is used.


DR# I’ve already addressed this above and previously. NLRI color identifies the route intent, which includes the intent for Anycast or Egress visibility.
Text in manageability/operational section provides relevant context for an operator. Everything else is as per already described procedures and discussions captured previously.

  F3-CAR-Q5: Status = Resolved
  ----------------------------

    Update Packing Observations are considered resolved. No additional text is
    needed.

DR# Thank you.

Regards,
-Dhananjaya


Pending Review: (Part II)
===============

- CAR-NLRI-Type-2
- CAR-NLRI-Type-1/2 interoperability
- VPN-CAR and VPN-CAR IP-Prefix
- Service family support
- Route target constraints and Scaling



Thanks,
-Nats-



Juniper Business Use Only
From: Susan Hares <[e-mail]>
Date: Wednesday, July 12, 2023 at 6:34 PM
To: [e-mail] <[e-mail]>
Cc: Natrajan Venkataraman <[e-mail]>
Subject: Re: [Idr] WG LC for draft-ietf-idr-bgp-car-02 (7/10/2023 to 7/24/2023)
[External Email. Be cautious of content]

Greetings:

I mentioned in the original call WG LC on Monday the following:

In parallel with this WG LC, I have asked Jeff Haas
(idr co-chair), Jie Dong (idr secretary), authors of
issues during adoption to review this version of the draft.
Directorate reviews have also been requested.

I would like to ask Natrajan (Nats)  to provide feedback for the from
Forum 3 of the adoption call which are specific CAR indicate
Whether their technical concerns have been resolved
In draft-ietf-idr-bgp-car-02?

1) F3-CAR-Issue-01: BGP-CAR Appendix A.7 – Anycast EP Scenario:
Author:  Natrajan Venkataraman [e-mail]<mailto:[e-mail]>

2) F3-CAR-Issue-2: BGP-CAR – Consensus on the need for resolution-schemes
Author:  Natrajan Venkataraman [e-mail]<mailto:[e-mail]>

3) F3-CAR-Issue-3:  CAR-Q3- Handling [of] LCM and Color Extended Communities
Author:  Natrajan Venkataraman [e-mail]<mailto:[e-mail]>

4) F3-CAR-Q4: BGP-CAR – Mis-Routing in Non-agreeing color-domains for Anycast EPs
Author:  Natrajan Venkataraman [e-mail]<mailto:[e-mail]>

5) F3-CAR-Q5: Update Packing Observations
Author:  Natrajan Venkataraman [e-mail]<mailto:[e-mail]>

Nats - please suggest any technical issues or changes
In text that may help this issue.

Cheerily, Sue Hares