Re: [Idr] BGP CAR - multiple color domains

["Nagarajah, Moses" <[e-mail]>]

Hi Dhananjaya,

I am Moses Nagarajah from Telstra networks. We have been following the BGP-CT and BGP-CAR drafts and the recent discussions, as we have use cases where we would need BGP to carry transport paths of different intents across domains.


Regarding your recent response in this thread,
<excerpt from Dhananjaya’s response>
Multiple operators have told us that they will manage and coordinate color allocation to keep design simple and avoid practices that create collisions. This is very similar to how they manage IP addresses already.
In our experience, especially with customers deploying Anycast / EPE scenarios, the predominant case will be that of a single color domain; i.e., multiple network domains having a consistent color-to-intent mapping. The typical usage of Anycast and EPE have already been described on the list. Both cases are automatically supported by CAR, and do not require use of LCM.

As Bruno and Jeff discussed in the thread below, a service such as Anycast must have some coordination when extended/stretched across admin/provider boundaries
</>

I disagree with the above statements from a practical deployment point of view.

Different operators/domains will be deploying / have already deployed colour aware transport tunnels/paths in their home domains primarily for the transport of local service flows.
And when the requirement arises for inter-domain intent awareness,  operators would expect (E, C) via N recursively resolves via (N, C) without conflicting the colours of existing home domain transport tunnels/paths.
As these deployments will happen organically over different timeframe, having a multilateral agreement won’t be practical. Otherwise, we need a universal syntax of  describing the intent to colour – which is also not practical.
Agreeing on a common unused IP address for Anycast or EPE is quite different from recolouring the existing home domain tunnels/paths to align with other operators. The former requires an agreement whereas the latter would require rework.

The original idea of LCM was to address the colour conflict problems which is evident that problem exists.
I don’t believe that we should ignore the problem because CAR needs to handle multipath/protection and route selection differently as highlighted by Nats below.

The term ‘typical usage’ in your response is subjective. We believe colour conflict can become common a problem in all the identified use cases and a technology framework should address all the possible /foreseen problems.

Also, consider the fact that we have conflicting QoS markings and treatments across different domains that grew organically even within the same organisation. When there is a need arises to interconnect these domains we harmonise and remap them to achieve the desired treatment. I believe, we will have such scenarios common in inter-domain transport class use cases in future.

Currently, we have separate networks for domestic and international and they are independent so as the TE policies. We intentionally maintain the autonomy and modularity for administrative purposes. When we need inter-domain intent awareness, we would need the same level of flexibility in the proposed solution.

I would also like to highlight, service provider networks usually have more meshed paths in the core and aggregation domains where more granular intents can be realised. However, the access network domain will have less number of paths ( either left or right in a ring / partial mesh / hub and spoke – in regional remote areas) where we would need only a few discrete transport classes / colours.
Hence, requirement for remapping of transport classes / colours within a single AS shouldn’t be considered as a corner case in my opinion.


Regards,
Moses Nagarajah
Network Architect
Transport & IP Evoloution - Networks & Infrastructure Engineering
[cid:image001.png@01D89243.023FA820]
Telstra

From: Dhananjaya Rao (dhrao) <dhrao=[e-mail]>
Sent: Thursday, 7 July 2022 12:38 AM
To: Natrajan Venkataraman <[e-mail]>; Kaliraj Vairavakkalai <[e-mail]>; DECRAENE Bruno INNOV/NET <[e-mail]>
Cc: [e-mail]
Subject: Re: [Idr] BGP CAR - multiple color domains

[External Email. Be cautious of content]


Hi Nats,

Apologies for the delay and long response.

Firstly, it is important to analyze whether there is a practical issue that must be addressed by the protocol or not.

Multiple operators have told us that they will manage and coordinate color allocation to keep design simple and avoid practices that create collisions. This is very similar to how they manage IP addresses already.

In our experience, especially with customers deploying Anycast / EPE scenarios, the predominant case will be that of a single color domain; i.e., multiple network domains having a consistent color-to-intent mapping.

The typical usage of Anycast and EPE have already been described on the list. Both cases are automatically supported by CAR, and do not require use of LCM.

E.g., https://mailarchive.ietf.org/arch/msg/idr/N-dymqn_c6xLck6FmRwJEXEqM5A/<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/idr/N-dymqn_c6xLck6FmRwJEXEqM5A/__;!!NEt6yMaO-gk!EG9-NwhFE-Plrb2k8CzVULTQ--OLXWb7T3aB32QBjZ5ff2MxSsn8uJRbM3LKWc18yzYivXLPfzhnaH95aVn-flHn36wpkCz5$>

We’ve also added an example to the CAR draft.

Now, for a case where such a shared service might possibly get stretched or extended across different provider/admin domains, a few options exist.

As Bruno and Jeff discussed in the thread below, a service such as Anycast must have some coordination when extended/stretched across admin/provider boundaries.

https://mailarchive.ietf.org/arch/msg/idr/0GIfHgtUCSF3Fu8m1pUdePSgaBc/<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/idr/0GIfHgtUCSF3Fu8m1pUdePSgaBc/__;!!NEt6yMaO-gk!EG9-NwhFE-Plrb2k8CzVULTQ--OLXWb7T3aB32QBjZ5ff2MxSsn8uJRbM3LKWc18yzYivXLPfzhnaH95aVn-flHn30Kyh4qk$>

The Anycast IP must be coordinated if it has to be same value. If they are different,, the color is anyway not an issue. But if the Anycast IP is coordinated, then it is totally reasonable to expect the Anycast Color to also be coordinated. And there’s a 32-bit space to do that from.

For a service such as EPE, which relies on advertising external peering IPs into the internal networks, it should be noted that the external IP is common only in the case the same external router peers with both the color domain border routers with the same IP address ( same loopback or possibly peering using a same interface IP on the same subnet). The general case where different routers or peering interfaces are used automatically ensure the IP is different.

For the specific case where the IP is same, again the better option would be for the two color domains to coordinate the use of a common color for the shared EPE service. But in the absence of this coordination, the design described in the BGP-LU EPE draft (https://datatracker.ietf.org/doc/html/draft-gredler-idr-bgplu-epe-14<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-gredler-idr-bgplu-epe-14__;!!NEt6yMaO-gk!EG9-NwhFE-Plrb2k8CzVULTQ--OLXWb7T3aB32QBjZ5ff2MxSsn8uJRbM3LKWc18yzYivXLPfzhnaH95aVn-flHn3xLzUs2C$>) can be applied.

That is, In case there is a color conflict for the exact EPE prefix(es), BGP ADD-PATH may be enabled between the EPE BRs and the BGP sessions towards the controller or to the ingress PEs that consume the EPE and service routes. It is very likely that these sessions already have ADD-PATH enabled to ensure the different EPE peer/peer-set instances are distributed without loss of any path visibility.

Hope this provides clarity on some of the options that may be used at present.

If requirements evolve and there is a compelling need for something more, we can address it.

Regards,
-Dhananjaya


From: Idr <[e-mail]<mailto:[e-mail]>> on behalf of Natrajan Venkataraman <natv=[e-mail]<mailto:natv=[e-mail]>>
Date: Thursday, March 31, 2022 at 1:38 AM
To: Kaliraj Vairavakkalai <kaliraj=[e-mail]<mailto:kaliraj=[e-mail]>>, DECRAENE Bruno INNOV/NET <[e-mail]<mailto:[e-mail]>>
Cc: "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>
Subject: Re: [Idr] BGP CAR - multiple color domains

Hi Bruno,

As Kaliraj just mentioned, I’m listing the following use-cases which we are currently deployed by customers and it would be really helpful to understand how CAR will handle these scenarios. I wasn’t able to find this in section 2.7 of the CAR draft. It would be good to describe how LCM will be use to merge the paths for ECMP/FRR.

UseCase 1:  EPE forwarding towards two AS domains with color disagreement for Multihomed Peer CE1
-------------------------------------------------------------------------------------------------

                                     Gold = 100 Bronze = 200
                            +--------[ASBR2     (AS2)     PE2]--+
    Gold = 100 Bronze = 200 |                                   |
    [PE1      (AS1)     ASBR1]                                 CE1 (1.1.1.1)
          INGRESS DOMAIN    |                                   |   Multihomed onto PE2,PE3
                              +--------[ASBR3     (AS3)     PE3]--+
                                     Gold = 200 Bronze = 100


UseCase 2:  Anycast forwarding to two domains with color disagreement
---------------------------------------------------------------------

    |--------ADMINISTRATIVE DOMAIN A ---------------------------|
                                     Gold = 100 Bronze = 200
                            +--------[ASBR2     (AS2)     PE2-1.1.1.1]
    Gold = 100 Bronze = 200 |
    [PE1      (AS1)     ASBR1]
          INGRESS DOMAIN    |
                              +--------[ASBR3     (AS3)     PE3-1.1.1.1]
                                     Gold = 200 Bronze = 100


Service Route (With color community): (PE1 Only)

  R1, Color:0:100
  R2, Color:0:200

CAR Routes: (PE1 and ASBR1)

      1.1.1.1:100

            Add-Path 1: 1.1.1.1:100          from AS2 (Gold)
            Add-Path 2: 1.1.1.1:100, LCM 200 from AS3 (Bronze)

      1.1.1.1:200

            Add-Path 1: 1.1.1.1:200          from AS2 (Gold)
            Add-Path 2: 1.1.1.1:200, LCM 100 from AS3 (Bronze)

Stating the following as per CAR draft:

      NLRI key                      = EP:Color
      Effective Resolution key      = EP:EffectiveColor (LCM overrides NLRI color)

      - CAR route multipath/protection creation is based on CAR NLRI key and
      - CAR route resolution is based on Effective Resolution Key.

Observed Behavior:

    - In PE1, using NLRI key misses ECMP/FRR computation for Ingress routes
    - In ASBR1, using NLRI key misses ECMP/FRR computation for Transit routes


A clarification of how these scenarios are handled in CAR would be really helpful.

-Nats-


From: Idr <[e-mail]<mailto:[e-mail]>> on behalf of Kaliraj Vairavakkalai <kaliraj=[e-mail]<mailto:kaliraj=[e-mail]>>
Date: Wednesday, March 30, 2022 at 11:50 AM
To: [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
Cc: [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
Subject: Re: [Idr] BGP CAR - multiple color domains
[External Email. Be cautious of content]

> If so, routes are originated on ingress Domain2/3 ASBR with color chosen by Domain 1 (C1)

Domain2,3 are egress-domains (traffic direction PE1->Peer1), which originate the Peer1/32 EPE route for the external peer.

So they would originate the route with color-namespace used in these respective domains only, not with color chose by domain1. E.g. Color C1 may have a different meaning in Domain3. So Domain3 cannot use C1 to describe the desired intent.

Thanks
Kaliraj
From: [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
Date: Wednesday, March 30, 2022 at 6:59 AM
To: Kaliraj Vairavakkalai <[e-mail]<mailto:[e-mail]>>
Cc: [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
Subject: RE: BGP CAR - multiple color domains
[External Email. Be cautious of content]

Hi Kaliraj,

Sorry for the delay.
Please see inline [Bruno]



Orange Restricted
From: Kaliraj Vairavakkalai <[e-mail]<mailto:[e-mail]>>
Sent: Wednesday, March 23, 2022 2:11 AM
To: DECRAENE Bruno INNOV/NET <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Subject: Re: BGP CAR - multiple color domains

Hi Bruno,

Please consider the following topology.

Two parallel Cores Domain2, Domain3. Domain1 having ingress node PE1. EBGP peer Peer1 multihomed to two core domains as shown below.

Traffic direction is PE1->Peer1. In each domain left side is ingress, right side is egress.

Usecase is: EPE forwarding towards Peer1.

Domain2, Domain3 egress ASBRs originate Peer1/32 route in the Transport-family (CAR for this discussion).
Similar to how we do with BGP-LU today (BGP-LU EPE1).

                                          Color C1
                                     +----------------+
                                     |  Core Domain2  |
                                    /+----------------+\
            +--------------------+/                     \+--------+
            |  Ingress  Domain1  |                       | Peer1  |
           PE1                   |                       +--------+
            +--------------------+\                     /
                Color  C1          \+-- --------------+/
                                    |   Core Domain3  |
                                    +-----------------+
                                          Color C2


[Bruno] We are probably having different point of view, but I’m having difficulties mapping your above example in deployments that I’m aware of.
I could imagine 2 options:
I) If Peer1/domain4  is using color I would imagine that it originates its own color route (Peer1, C4) that would be distributed in domain2 and domain3 which would resolve C4 using their own local color (resp. C1, C2). Cf https://datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-03#appendix-B.2<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-03*appendix-B.2__;Iw!!NEt6yMaO-gk!TqcthKTo6nLwSNq051Xus2lTb5O9Qv-AbGx5FWy636W50NJYBn9exG92Uqf92kYV$>
II) Peer1/domain4  is not using color. Color route Peer1 is originated by Domain2 and Domain3 for domain1 to use as per its request. If so, routes are originated on ingress Domain2/3 ASBR with color chosen by Domain 1 (C1). Again, Domain2 and Domain3 would resolve next-hop using their own color schema.

You seem describe a third option where Peer1/domain4 is not using color but domain2/3 would need to originate color routes on the egress domain2/3 ASBR. But I’m not seeing why/what for.

Thanks for clarifying as I’m probably missing something (hopefully not so obvious).

--Bruno


Domain1, Domain2 use color C1 value to indicate a certain Transport-class (eg. 'high-bandwidth'). Domain3 uses C2 for same.

Now, the ingress ASBRs in Domain3 will use LCM(Color=C2) in (Peer1, C2) advertisement towards Domain1, such that Domain1
will remap to LCM(C1). So Domain1 egress ASBR will have the following routes in the BGP-RIB for CAR family:

        (Peer1, C1)
        (Peer1, C2), LCM(C1).

As you can see, Multipath/Protection can no longer be computed on the BGP NLRI prefix (Peer1, Cx). It needs to be computed
based on (Peer1, Effective-color C1). This is what I was trying to point out.

Further, Ingress PE1 will have the same information at transport-layer. And when resolving a Service-route received with
Nexthop Peer1, Color:0:C1, it cannot use just the BGP-NLRI prefix (Peer1, C1) as the resolving route. Doing so will miss
the Multipath/Protection. It will need to resolve over the (Peer1, Effective color C1). So that the service prefix gets
Multipath/Protection towards the two domains Domain2, Domain3.

Similar usecase can be constructed for Anycast EP in Domain2, Domain3 also.

So, though one may argue that EPE and Anycast Endpoints are not the common-case, I strongly believe such deployment scenarios
should be supported. Thanks to Ben for bringing up EPE as a use-case customers are interested in.

What we think of as corner case or may not happen - will certainly happen in the field. Nature has its way! Murphys Law!. :)

Thanks
Kaliraj

1 BGP-LU EPE: https://datatracker.ietf.org/doc/html/draft-gredler-idr-bgplu-epe-14<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-gredler-idr-bgplu-epe-14__;!!NEt6yMaO-gk!TqcthKTo6nLwSNq051Xus2lTb5O9Qv-AbGx5FWy636W50NJYBn9exG92UoZiuIB4$>

From: Idr <[e-mail]<mailto:[e-mail]>> on behalf of [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
Date: Tuesday, March 22, 2022 at 10:45 AM
To: [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
Subject: [Idr] BGP CAR - multiple color domains
[External Email. Be cautious of content]

Hi BGP CT authors,

As the subject is a bit vast, I’d like to better understand your operational concern with multiple colors domains.

At your convenience, I think that three texts could be used to support our discussion

  1.  Please feel free to explain the issue your seeing with you own text.
  2.  This 1 page is probably a good start https://datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-03#section-2.8<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-03*section-2.8__;Iw!!NEt6yMaO-gk!VQI-5zbHY7CE6clhUhOhP9Z_PljSz_MeeS11L5-pq_RckcjiDJdGhd0N2atrcsQQ$>
  3.  I’ve tried to describe the whole route journey in the below text using an example from a requirement document https://datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-problem-statement#section-1.2.9<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-problem-statement*section-1.2.9__;Iw!!NEt6yMaO-gk!VQI-5zbHY7CE6clhUhOhP9Z_PljSz_MeeS11L5-pq_RckcjiDJdGhd0N2ZKQ6JLD$> and you can raise the issue when you see it.


So below is option 3 text. It’s much longer and painful so if “2” is good enough you could skip the below text.

Please note that I’ll use a terminology from https://datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-problem-statement#section-1.2<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-problem-statement*section-1.2__;Iw!!NEt6yMaO-gk!VQI-5zbHY7CE6clhUhOhP9Z_PljSz_MeeS11L5-pq_RckcjiDJdGhd0N2QNUDSgu$> and that colored route are not to be confused with color-aware route.

Let’s consider option C with 2 domains:


     +----------------+  +----------------+
     |            E3  |  |                | V/v with C1
     |----+          +----+          +----|/
     | E1 |          | N2 |          | E2 |\
     |----+          +----+          +----| W/w with C2
     |                |  |                |
     |    Domain 1    |  |    Domain 3    |
     +----------------+  +--- ------------+


   *  Service routes MUST be colored using BGP Color Extended-Community
      to request intent

      -  V/v via E, colored with C

   *  Colored service routes MUST be automatically steered on an
      appropriate color-aware path

      -  V/v via E with C is steered via (E, C)


First color resolution seem the above one.
A priori the color from the VPN route (V/v via E with C) is the same as the color from the transport route (E, C) as both are chosen by the Egress domain (Domain 3).
Agreed or am I missing something?

Now in domain 1 and let’s assume that domain 1 uses color C to mean “high bandwidth” while domain 3 use color C to mean “low delay”
First, let’s notice that key is (E,C) so we are not going to mix/compare color C between (E2, C) and (E3, C). We are interested in different colors to reach a specific destination E, and all colors for that destination are consistent (defined in the domain of E). So I don’t see any issue with ECMP or protection that have been raised during the meeting.


Let’s continue with next steps



   *  Color-aware routes MAY resolve recursively via other color-aware

      routes



      -  (E, C) via N recursively resolves via (N, C)


Here I can see the mismatch as C from (E,C) from domain 3 while C from (N,C) is from domain 1 and hence may not be directly comparable without a mapping. So mapping is needed (I think all solutions will require a (re)mapping).
Except for this remapping, is there a big issue such as confusion?

Coming back to the remapping, this seems to depend on the internal routing solution used in Domain 1:
- If FlexAlgo, N2 can probably do the mapping : N2, C1 is advertised in Domain 1 FA associated with the right meaning (e.g. low delay)
- worst case we need to re-color i.e. express that the color-aware route (E,C) need to be resolved using a specific color. Personally, I’m not sure why the same BGP Color Extended community can’t be reused just like https://datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-problem-statement#section-1.2.3<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-problem-statement*section-1.2.3__;Iw!!NEt6yMaO-gk!VQI-5zbHY7CE6clhUhOhP9Z_PljSz_MeeS11L5-pq_RckcjiDJdGhd0N2X3yl744$>

but that’s a detail and defining a different community Local-Color-Mapping-Extended-Community https://datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-03#section-2.8<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-dskc-bess-bgp-car-03*section-2.8__;Iw!!NEt6yMaO-gk!VQI-5zbHY7CE6clhUhOhP9Z_PljSz_MeeS11L5-pq_RckcjiDJdGhd0N2atrcsQQ$>  which seems to indicate the same thing (the color of the color-aware route to use when resolution is done).

That’s all for the route journey. Hopefully all that text will be useful to pinpoint the issue that you have in mind.

--Bruno

_________________________________________________________________________________________________________________________



Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.



This message and its attachments may contain confidential or privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.


Juniper Business Use Only

_________________________________________________________________________________________________________________________



Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.



This message and its attachments may contain confidential or privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.


Juniper Business Use Only


Juniper Business Use Only


Juniper Business Use Only


Juniper Business Use Only