Re: [Gen-art] [Last-Call] Genart last call review of draft-ietf-bess-datacenter-gateway-10

[John E Drake <[e-mail]>]

Excellent, thanks so much for your help on this.

Yours Irrespectively,

John


Juniper Business Use Only

> -----Original Message-----
> From: Gyan Mishra <[e-mail]>
> Sent: Tuesday, May 18, 2021 4:28 PM
> To: Lars Eggert <[e-mail]>
> Cc: General Area Review Team <[e-mail]>; [e-mail]; draft-ietf-
> [e-mail]; [e-mail]
> Subject: Re: [Last-Call] Genart last call review of draft-ietf-bess-datacenter-
> gateway-10
> 
> [External Email. Be cautious of content]
> 
> 
> Hi Lars’s  & DC Gateway authors
> 
> I will be responding back today to the Gen-Art original email I sent with final
> comments and hope the final comments will help improve the document.
> 
>     I will also address the comments from John Scudder related to GW failover as
> well as Alvaro’s comments related to tunnel encapsulation attribute BGP prefix
> sid Sub-TLV limitations.  Also will add new text recommendations related to RFC
> 2119 MUST / SHOULD language to help improve the document.
> 
> 
> Thank you
> 
> Gyan
> On Tue, May 18, 2021 at 3:31 AM Lars Eggert <[e-mail]> wrote:
> 
> > Gyan, thank you for your review and thank you all for the following
> > discussion. I have entered a No Objection ballot for this document
> > based on the current status of the discussion.
> >
> > Lars
> >
> >
> > > On 2021-4-29, at 8:46, Gyan Mishra via Datatracker
> > > <[e-mail]>
> > wrote:
> > >
> > > Reviewer: Gyan Mishra
> > > Review result: Not Ready
> > >
> > > I am the assigned Gen-ART reviewer for this draft. The General Area
> > > Review Team (Gen-ART) reviews all IETF documents being processed by
> > > the IESG for the IETF Chair.  Please treat these comments just like
> > > any other last call comments.
> > >
> > > For more information, please see the FAQ at
> > >
> > >
> <https://urldefense.com/v3/__https://trac.ietf.org/trac/gen/wiki/GenArtfaq__;
> !!NEt6yMaO-gk!RIcJvmiBoFFiuLezPbzRuUXybG_QHD8PujD7pROBUPot5dc9nX-
> rMTiD1THCYZA$ >.
> > >
> > > Document: draft-ietf-bess-datacenter-gateway-??
> > > Reviewer: Gyan Mishra
> > > Review Date: 2021-04-28
> > > IETF LC End Date: 2021-04-29
> > > IESG Telechat date: Not scheduled for a telechat
> > >
> > > Summary:
> > >   This document defines a mechanism using the BGP Tunnel Encapsulation
> > >   attribute to allow each gateway router to advertise the routes to the
> > >   prefixes in the Segment Routing domains to which it provides access,
> > >   and also to advertise on behalf of each other gateway to the same
> > >   Segment Routing domain.
> > >
> > > This draft needs to provide some more clarity as far as the use case
> > > and
> > where
> > > this would as well as how it would be used and implemented.  From
> > reading the
> > > specification it appears there are some technical gaps that exist.
> > > There
> > are
> > > some major issues with this draft. I don’t think this draft is ready yet.
> > >
> > > Major issues:
> > >
> > > Abstract comments:
> > > It is mentioned that the use of Segment Routing within the Data Center.
> > Is
> > > that a requirement for this specification to work as this is
> > > mentioned throughout the draft?  Technically I would think the
> > > concept of the
> > discovery
> > > of the gateways is feasible without the requirement of SR within the
> > > Data Center.
> > >
> > > The concept of load balancing is a bigger issue brought up in this
> > > draft
> > as the
> > > problem statement and what this draft is trying to solve which I
> > > will
> > address
> > > in the introduction comments.
> > >
> > > Introduction comments:
> > > In the introduction the use case is expanded much further to any
> > functional
> > > edge AS verbiage below.
> > >
> > > OLD
> > >
> > >   “SR may also be operated in other domains, such as access networks.
> > >   Those domains also need to be connected across backbone networks
> > >   through gateways.  For illustrative purposes, consider the Ingress
> > >   and Egress SR Domains shown in Figure 1 as separate ASes.  The
> > >   various ASes that provide connectivity between the Ingress and Egress
> > >   Domains could each be constructed differently and use different
> > >   technologies such as IP, MPLS with global table routing native BGP to
> > >   the edge, MPLS IP VPN, SR-MPLS IP VPN, or SRv6 IP VPN”
> > >
> > > This paragraph expands the use case to any ingress or egress stub
> > > domain
> > Data
> > > Center, Access or any.  If that is the case should the draft name
> > > change
> > to
> > > maybe a “stub edge domain services discovery”.  As this draft can be
> > used for
> > > any I would not preclude any use case and make the GW discovery open
> > > to
> > be used
> > > for any service GW edge function and change the draft name to
> > > something
> > more
> > > appropriate.
> > >
> > > This paragraph also states for illustrative purposes which is fine
> > > but
> > then it
> > > expands the overlay/underlay use cases. I believe this use case can
> > > only
> > be
> > > used for any technology that has an overlay/underlay which would
> > preclude any
> > > use case with just an underlay global table routing such as what is
> > mentioned
> > > “IP, MPLS with global table routing native BGP to the edge.  The IP
> > > or
> > global
> > > table routing would be an issue as this specification requires
> > > setting a
> > RT and
> > > an export/import RT policy for the discover of routes advertised by
> > > the
> > GWs.
> > > As I don’t think this solution from what I can tell would work
> > technically for
> > > global table routing I will update the above paragraph to preclude
> > global table
> > > routing.  We can add back in we can figure that out but I don’t
> > > think any public or private operator would change from global table
> > > carrying all
> > BGP
> > > prefixes in the underlay now drastic change to VPN overlay pushing
> > > all
> > the
> > > any-any prefixes into the overlay as that would be a prerequisite to
> > > be
> > able to
> > > use this draft.
> > >
> > >> From this point forward I am going to assume we are using VPN
> > >> overlay
> > > technology such as SR or MPLS.
> > >
> > > NEW
> > >
> > >   “SR may also be operated in other domains, such as access networks.
> > >   Those domains also need to be connected across backbone networks
> > >   through gateways.  For illustrative purposes, consider the Ingress
> > >   and Egress SR Domains shown in Figure 1 as separate ASes.  The
> > >   various ASs that provide connectivity between the Ingress and Egress
> > >   Domains could be two as shown in Figure-1 or could be many more as
> > exists
> > >   with the public internet use case, and each may be constructed
> > differently
> > >   and use different technologies such as MPLS IP VPN, SR-MPLS IP
> > > VPN, or
> > SRv6
> > >   IP VPN” with a “BGP Free” Core.
> > >
> > > This may work without “BGP Free” core but I think to simplify the
> > > design complexity I think constraining to “BGP Free” core transport layer.
> > SR-TE path
> > > steering as well gets much more complicated if all P routers are
> > > running
> > BGP as
> > > well. I think in this example we can even explicitly say this
> > > example
> > shows the
> > > public internet as that would be one of the primary use cases.
> > >
> > > This paragraph is confusing to the reader
> > >
> > > As a precursor to this paragraph I think it maybe a good idea to
> > > state
> > that we
> > > are talking global table IP only routing or VPN overlay technology
> > > with
> > SR/MPLS
> > > underlay transport.  That will make this section much easier to
> > understand.
> > >
> > > Figure 1 drawing you should give a AS number to both the ingress
> > > domain
> > and
> > > egress domain so the reader does not have to make assumptions if it
> > > iBGP
> > or
> > > eBGP connected to the egress or ingress domain and state eBGP in the
> > > text below.  Lets also call the intermediate ASNs in the middle as
> > > depicted
> > in the
> > > diagram could be 2 as shown illustratively but could be many
> > > operator
> > domains
> > > such as in the case of traversing the public internet.   In the drawing
> > I would
> > > replace ASBR for PE as per this solution as I am stating it has to
> > > be a
> > VPN
> > > overlay paradigm and not global routing.  Also in the VPN overlay
> > scenario when
> > > you are doing any type of inter-as peering the inter-AS peering is
> > > almost always between PE’s and not a separate dedicated device
> > > serving a special “ASBR-ASBR” function as the PE is acting as the
> > > border node providing the “ASBR” type function.  So in the re-write
> > > I am assuming the drawing has
> > been
> > > updated changing ASBR to  PE.  Lets give each node a number so that
> > > we
> > can be
> > > clear in the text exactly what node we are referring to.  In the
> > > drawing
> > please
> > > update that GW1 peers to PE1 and GW2 peers to PE2 and GW3 peers to PE3.
> > GW3
> > > also peers to GW4 and GW2 peers  to GW5 which GW4 and GW5 are part
> > > of
> > AS3.  In
> > > the AS1-AS2 peering  top peer would be PE6 peers to PE8 and bottom
> > > peer
> > PE7
> > > peers to PE9.  So PE6 and PE7 are in AS1 and PE8 and PE9 are in AS2.
> > > I
> > made
> > > the bottom to ASBRs in AS3 for the selective deterministic load
> > balancing now
> > > calling them GW4 and GW5 used later in the problem statement.
> > >
> > > One major problem with this problem statement description is that it
> > > is incorrect as far as GW load balancing that it does not work today
> > > in the topology given in Figure-1.  The function of edge GW load
> > > balancing is
> > based on
> > > the iBGP path tie breaker lowest common denominator in the BGP path
> > selection
> > > which is lowest IGP underlay metric and as long as the metric is
> > > equal
> > and you
> > > have iBGP multipath enabled  you now can load balance to egress PE1
> > > and
> > PE2
> > > endpoints. So in this case flows coming from AS1 into AS2 hit a P
> > intermediate
> > > router which has iBGP multipath enabled and has lets say equal cost
> > > for
> > route
> > > to the next hop attribute assuming next-hop-self is set so the cost
> > > to
> > > loopback0 on PE1 and cost to loopback0 on PE2 is lets say 10, so now
> > > you
> > have a
> > > BGP multipath.  What is required though is the RD has to be unique
> > > in a
> > “BGP
> > > Free” core RR environment where all PE’s route-reflector-clients
> > > peer to
> > the RR
> > > and for all the paths that are advertised to the RR to be reflected
> > > to
> > all the
> > > egress PE edges the RD must be unique for the RR to reflect all paths.
> > BGP
> > > add-paths is only used if you have Primary and Backup routing setup
> > > where
> > > PE1-GW1 has a 0x prepend and PE2-GW2 has 1x prepend so now with BGP
> > add-paths
> > > along with BGP PIC Edge you now have a edge pre-programmed backup
> path.
> > So the
> > > add-paths is not necessarily something that helps for load balancing
> > > and
> > is in
> > > fact orthogonal to load balancing as it for Primary / Backup routing
> > > and
> > not
> > > Active/Active load balancing routing where load balancing with VPN
> > overlay is
> > > simply achieved with unique RD per PE and iBGP multipath and equal
> > > cost
> > paths
> > > to the underlay recursive IGP learned next-hop-attribute in this
> > > case
> > the PE
> > > loopback 0 per the next hop rewrite via “next-hop-sellf” done on the
> > PE-RR
> > > peering in a standard VPN overlay topology.   As far as load balancing
> > being
> > > accomplished in the underlay what I have stated is independent of
> > > SR-TE
> > however
> > > with SR-TE candidate path the load balancing ECMP spray to egress PE
> > egress GW
> > > AS can also happen as well with prefix-sid.
> > >
> > > OLD
> > >   Suppose that there are two gateways, GW1 and GW2 as shown in
> > >   Figure 1, for a given egress SR domain and that they each advertise a
> > >   route to prefix X which is located within the egress SR domain with
> > >   each setting itself as next hop.  One might think that the GWs for X
> > >   could be inferred from the routes' next hop fields, but typically it
> > >   is not the case that both routes get distributed across the backbone:
> > >   rather only the best route, as selected by BGP, is distributed.  This
> > >   precludes load balancing flows across both GWs.
> > >
> > > I am rewriting the text in the NEW as there is some discrepancy in
> > > the
> > routes
> > > being distributed across the backbone and what gets distributed.  So
> > > I am completely re-writing to make it more clear what we are trying
> > > to state
> > here as
> > > the text appears technically to be incorrect.  To help state the
> > > flow
> > will use
> > > the BGP route flow to help depict the routing and try to get to the
> > problem
> > > statement we are trying to portray.
> > >
> > > NEW
> > >
> > >   Suppose that there are two gateways, GW1 and GW2 as shown in
> > >   Figure 1, for a given egress SR domain and each gateway advertises
> > > via
> > EBGP
> > >   a VPN prefix X to AS2 core domain via EBGP with underlay next hop
> > > set
> > to GW1
> > >   or GW2. In this case we are Active / Active load balancing with
> > > PE1
> > and PE2
> > >   receives the VPN prefix and advertised the VPN prefix X into the
> > domain with
> > >   next-hop-self set on the PE-RR peering to the PE’s loopback0.  The
> > > P
> > routers
> > >   within the domain have ECMP path with IGP metric tie to the egress
> > > PE1
> > and
> > >   egress PE2 for VPN Prefix X learned from GW1 and GW2. SR-TE path
> > > can
> > now be
> > >   stitched from GW3 to PE3 SR-TE Segment-1 to PE3 to PE6 and PE7
> > Segment-2 to
> > >   PE8 and PE9 to Egress Domain via PE1 and PE2 to GW1 and GW2.  In
> > > this
> > case
> > >   however we don’t want the traffic to be steered via SR-TE Load
> > balanced via
> > >   ingress GW3 and want to take GW3 out of rotation and load balance
> > traffic to
> > >   GW4 and GW5 instead.
> > >
> > > **Text above provides the updated selective deterministic gateway
> > steering
> > > described below to achieve the goal.  I think that may have been the
> > intent of
> > > the authors and I am just making it more clear**
> > >
> > > As for problem statement as GW load balancing can occur in the
> > > underlay
> > as
> > > stated easily that is not the problem.
> > >
> > > In my mind I am thinking the problem statement that we want to
> > > describe
> > in both
> > > the Abstract and Introduction is not vanilla simple gateway load
> > balancing but
> > > rather a predictable deterministic method of selecting gateways to
> > > be
> > used that
> > > is each VPN prefix now has a descriptor attached -  tunnel
> > > encapsulation attribute which contains multiple TLVs one or more for
> > > each “selected
> > gateway”
> > > with each tunnel TLV contains an egress tunnel endpoint sub-tlv that
> > identifies
> > > the gateway for the tunnel.  Maybe we can have in the sub-tlv a
> > > priority
> > field
> > > for pecking order preference of which GWs are pushed up into the GW hash
> > > selected for the SR-ERO path to be stitched end to end.   So lets say
> > you had
> > > 10 GWs and you break them up into 2 tiers or multi tiers and have
> > > maybe
> > gateway
> > > 1-5 are primary and 6-10 are backup and that could be do to various
> > reasons so
> > > you can basically pick and choose based on priority which GW that
> > > gets
> > added to
> > > the GW hash.
> > >
> > > I have some feedback and comments on the solution and how best to
> > > write
> > the
> > > verbiage to make it more clear to the reader.
> > >
> > > I think in the solution as far s the RT to attach for the GW auto
> > discovery.
> > > So with this new RT we are essentially creating a new VPN RIB that
> > > has
> > prefixes
> > > from all the selected gateways that are discovered from the tunnel
> > > encapsulation attribute TLV.
> > >
> > > In the text here what is really confusing is if the tunnel
> > > encapsulation attribute is being attached to the underlay recursive route to
> next hop
> > > attribute or the VPN overlay prefix.   So the reason I am thinking it is
> > being
> > > attached to the VPN overlay prefix and not the underlay next hop
> > attribute is
> > > how would you now create another transport RIB and if you are
> > > creating a
> > new
> > > transport RIB there is already a draft defined by Kaliraj
> > > Vairavakkalai
> > or
> > > BGP-LU SAFI 4 labeled unicast that exits today to advertise next
> > > hops
> > between
> > > domains for an end to end LSP load balanced path.
> > >
> > >
> > https://urldefense.com/v3/__https://tools.ietf.org/html/draft-kaliraj-
> > idr-bgp-classful-transport-planes-07__;!!NEt6yMaO-gk!RIcJvmiBoFFiuLezP
> > bzRuUXybG_QHD8PujD7pROBUPot5dc9nX-rMTiD7W4i_nA$
> > >
> > > IANA code point below
> > > 76      Classful-Transport SAFI
> > > [draft-kaliraj-idr-bgp-classful-transport-planes-00]
> > >
> > > Also in line with CT another option is BGP-LU SAFI 4 to import the
> > loopbacks
> > > between domains which is the next hop attribute to be advertised
> > > into
> > the core
> > > end to end LSP.  So the BGP-LU SAFI  RIB could be used for the next
> > > GW
> > next hop
> > > advertisement between domains so that there is visibility of all the
> > egress PE
> > > loopback0 between domains.   So you can either stitch the LSP segmented
> > LSP
> > > like inter-as option-b SR-TE stitched and use nex-hop self PE-RR
> > > next-hop rewrite on each of the PEs within the internet domain or
> > > you could
> > import all
> > > the PE loopback from all ingress and egress domains into the
> > > internet
> > domain
> > > similar to inter-as opt-c create end to end LSP instantiate an end
> > > to
> > end SR-TE
> > > path.
> > >
> > > Maybe you could attach the RT tunnel encapsulation attribute tunnel
> > > tlv endpoint tlv to the VPN overlay prefix.  Not sure how that would
> > > be
> > beneficial
> > > the underlay steers the VPN overlay.
> > >
> > > So maybe you could couple the VPN overlay new GW RIB RT to the
> > > transport Underlay CT CLAS RIB or BGP-LU RIB coupling  may have some
> > > benefit but
> > that
> > > would have to be investigated but I think is out of scope of the
> > > goals
> > of this
> > > draft.
> > >
> > > I think we first have to figure out the goal and purpose of this
> > > draft
> > by the
> > > authors and how the GW discovery should work in light of the CT
> > > class CT
> > RIB
> > > AFI/SAFI codepoint draft that exists today as well as the BGP-LU
> > > option
> > for
> > > next hop advertisement within the internet domain.
> > >
> > > Section 3 comments
> > >
> > >      “Each GW is configured with an identifier for the SR domain.  That
> > >      identifier is common across all GWs to the domain (i.e., the same
> > >      identifier is used by all GWs to the same SR domain), and unique
> > >      across all SR domains that are connected (i.e., across all GWs to
> > >      all SR domains that are interconnected).
> > >
> > > **No issues with the above**
> > >
> > >      A route target ([RFC4360]) is attached to each GW's auto-discovery
> > >      route and has its value set to the SR domain identifier.
> > >
> > > **So here if the RT is attached to the GW auto-discovery route we
> > > need
> > to state
> > > is that the underlay route and that the PE does a next-hop-self
> > > rewrite
> > of the
> > > eBGP link to the BGP egress domain next hop to the loopback0 so the
> > > GW
> > next hop
> > > that we are tracking of all the ingress and egress PE domains is the
> > egress and
> > > ingress PE loopback0.**
> > >
> > >      Each GW constructs an import filtering rule to import any route
> > >      that carries a route target with the same SR domain identifier
> > >      that the GW itself uses.  This means that only these GWs will
> > >      import those routes, and that all GWs to the same SR domain will
> > >      import each other's routes and will learn (auto-discover) the
> > >      current set of active GWs for the SR domain.”
> > >
> > > **So if this is the case and we are tracking the underlay RIB and
> > > attach
> > a
> > > route target to all the ingress PE & P next hops which is loopback0
> > > =
> > this is
> > > literally identical to BGP-LU importing all the loopbacks between
> > domains or
> > > using CT class** There is no need for this feature to use the tunnel
> > > encapsulation attribute.  I am not following why you would not use
> > BGP-LU or CT
> > > clas RIB.**
> > >
> > >   “To avoid the side effect of applying the Tunnel Encapsulation
> > >   attribute to any packet that is addressed to the GW itself, the GW
> > >   SHOULD use a different loopback address for packets intended for it.”
> > >
> > > **I don’t understand this statement as the next hop is the ingress
> > > and
> > egress
> > > PE loopback0 that is the next hop being tracked for the gateway load
> > balancing.
> > > The GW device subnet between the GW and PE is not advertised into
> > > the
> > internet
> > > domain as we do next-hop-self on the PE PE-RR iBGP peering and so
> > > the GW
> > to PE
> > > subnet is not advertised.**   Looking at it a second time I think we are
> > > thinking here BGP-LU inter-as opt c style import of loops between
> > domains and
> > > so instead of importing the loop0 which carries all packets on the
> > > GW
> > device
> > > use a different loopback GW1 so it does not carry the FEC of all
> > > BAU
> > packets
> > > similar concept utilized in RSVP-TE to VPN mapping "per-vrf TE"
> > > concept