[Idr] 答复: draft-xu-idr-bgp-route-broker-01.txt

["[e-mail]" <[e-mail]>]

Hi Robert,

发件人: Robert Raszuk <[e-mail]>
日期: 星期二, 2023年8月1日 23:29
收件人: [e-mail] <[e-mail]>
抄送: Srihari Sangli <[e-mail]>, Shraddha Hegde <[e-mail]>, idr@ietf. org <[e-mail]>
主题: Re: draft-xu-idr-bgp-route-broker-01.txt
Hi Xiaohu,

Few follow up comments, but stay tuned for more :)

[Xiaohu] It actually leverages the route target constrained distribution mechanism😊

Cool. You have not mentioned it ... nor it is part of your reference section.

[Xiaohu] Good suggestion. I will add it as a reference in the next version.

[Xiaohu] If there was no permanent session between the publishers and subscribers, it seems that it’s hard to update the route tables on virtual PE routers in a real-time manner.

Not really. That is the whole idea of "subscription" to information.

[Xiaohu] AKAIK, even in the pub/sub system, it’s desirable for the brokers/exchanges to establish persistent connections with the publishers. See the following text quoted from RabbitMQ tutorial: (https://www.rabbitmq.com/publishers.html#:~:text=Publishers%20are%20often%20long%20lived%3A%20that%20is%2C%20throughout,long%20as%20their%20connection%20or%20even%20application%20runs.):

Publishers are often long lived: that is, throughout the lifetime of a publisher it publishes multiple messages. Opening a connection or channel (session) to publish a single message is not optimal. Publishers usually open their connection(s) during application startup. They often would live as long as their connection or even application runs.

[Xiaohu] AFAIK, RobbitMQ (https://www.rabbitmq.com/connections.html), as one of the most popular large-scale pub/sub systems, are built on TCP😊 Hence, the root cause of the scaling issue as mentioned above should not be TCP.

That is completely not my point. When I referred to TCP I meant BGP-4 TCP.

TCP is a great protocol ... HTTP/1.1 or HTTP/2 runs on TCP - but the crux of the matter is that the sessions are short lived. There is no (with few exceptions of active data streaming) keep the sessions for hours, days, months or years like in BGP.

[Xiaohu] In the data center network virtualization scenario where two-level RRs are deployed, the top-level RRs act as publishers, the bottom-level RRs, referred to as BGP route brokers, act as exchanges, and the virtual PE routers act as both publishers and subscribers. As a result, there should be long-lived connections between BGP route brokers and top-level RRs, also between BGP route brokers and virtual PE routers.


 #3 - You talk about overlay - but if there is overlay then usually there is also underlay. Dependency on underlay to detect nodes going down should be sufficient to trigger overlay invalidation.

[Xiaohu] Assume there are tens of thousands vPE routers, route aggregation on the underlay is a good practice. Furthermore, consider the VM migration operation or the frequent container creation/deletion operation, a proactive route distribution mechanism should be a must.

Even if you have reachability aggregation in the underlay there are known ways to detect the liveness of the peer.

[Xiaohu] I agree. BFD is a concrete example😊 However, the liveness of the peer doesn’t mean the routes originated from that peer are alive, think about the VM migration scenario.

#4 - You talk about PEs. But in modern L3 fabric MSDCs it is compute nodes (or actually even smartNICs) which are acting as PEs to their local tenants.

[Xiaohu] Agree. That’s the reason why the virtual PE concept is mentioned in the draft.

Ok. But also just think how your proposal works when vPEs or PEs are mobile nodes ....

[Xiaohu] Interesting, I have not yet thought about that case. Could you please say more about the scenarios where vPEs or Pes are mobile nodes?

So how about you think about DNS model or LISP mapping plane model instead here to distribuite overlay addresses to underlay next hop mappings ?

[Xiaohu] By using DNS or LISP, how could the virtual PE routers obtain the VPN route in a real-time maner?

By subscribing to interested information.

With HTTP/3 session establishment between known peers takes 0 RTT !  Just a hint :)

[Xiaohu] Interesting， this hint reminds me of the usage of XMPP as an alternative to BGP/Netconf in TF (also named as opencontrail)😊


So to summarize it is evident that you are really concerned about the number of TCP based BGP4 sessions.
Number of routes will still hit you as top level Route Reflectors (or Route Servers) will likely get requests to carry all VPN memberships.

[Xiaohu] there is no need to have any one top-level route reflector maintain all the VPN routes for all the VPNs supported by the data center. The scalability issue lies on the bottom-level route reflectors.

With dynamic VPN membership signalling you do not know that.

[Xiaohu] There is some details which are different from the RTC approach. Please refer to the following text quoted from section 3, “Top-level route reflectors, referred to as route servers, advertise route target membership information according to the preconfigured block of Route Targets. As such, route brokers know the VPNs associated with each of them. The route target membership information received form route servers SHOULD NOT be reflected by route brokers to any other iBGP peers further……The route target membership information received from route broker clients would be deemed by route brokers as an implicit route request for all the VPN routes for the VPNs associated to the corresponding route targets, and only need to be reflected towards the corresponding route servers which are associated with the VPNs associated with the advertised route targets.¶<https://datatracker.ietf.org/doc/html/draft-xu-idr-bgp-route-broker#section-3-2>”

See the other problem you are facing here is that today all L3VPN or EVPN deployments use static (management station driven) RT assignments.

Sure you can spread RT blocks between RRs, but this is what RTC has been suggesting nearly 20 years back :)

[Xiaohu] I’m very willing to reuse or leverage the existing mechanisms such as RTC.

Your proposal does mitigate the former by introducing the notion of "BGP Broker" - but instead of making half step I would encourage you to think further and use other then BGP protocol distribution mechanism.

[Xiaohu] Since BGP has been used as an overlay routing protocol by some open-source or commercial SDN solutions, such as TungsenFabric, it seems worthwhile to consider how to further improve the scalability of the BGP-based IP VPN technology when deploying it in hyperscale data center network virtualization environments.

I was sure you are going to quote Contrail here :)

Sure I am all for improving what's inefficient. But sometimes I like to also think outside of the box first. Will share a note which I have started to write about this problem space sometime back - but now your draft has fueled the new energy to finish it :)

[Xiaohu] It’s a real problem in the real world and therefore it’s worthwhile to deal with it😊

Best regards，
Xiaohu

Cheers,
Robert



Or if you like, go for using either hierarchy of route brokers with session less overlay transport.
[Xiaohu] I have not got your point of using a hierarchy of route brokers. Could you please say more?


Last but not least I am really surprised that with such proposal you are also not suggesting to use notion of aggregate withdraws as defined in https://www.ietf.org/archive/id/draft-raszuk-aggr-withdraw-00.txt

[Xiaohu] I have not yet found how to use the notion of aggregation withdraws to address the scalability issues as mentioned above. Let’s me think it further. However, it does use some similar trick (see Section 6) as yours.

Thanks a lot for your comments and suggestions again😊

Kind regards,
Robert