Re: [MBONED] [Msr6] MSR6 BOF 3rd Issue Category: More details are requested about the large scale use cases, including issue 8-11

[Toerless Eckert <[e-mail]>]

On Fri, Oct 28, 2022 at 09:52:38AM -0700, Dino Farinacci wrote:
> It sounds like a compromise could be to use the bitfield concept introduced by BIER and put those in an IPv6 packet. But the bits describe the oif-list like BIER does and not the receiver hosts. Then you bring the scale down to "the number of interfaces that lead to the edges of this domain".

Dino:

I think MSR6 has two orthogonal set of challenges/solutions:

One is scalability. Thats about better source-route/tree encoding than
flat bitstrings. e.g.: draft-eckert-bier-rbs. This is independent IMHO
of what encap to choose, e.g.: RFC8296 or something meant to
support IPv6 only networks, compliant with RFC8200 source routing like
SRH.

The other is exactly such an IPv6 encapsulation. Like we already have
two for unicast - SRH and RFC6554. IMHO, to really allow stateless
source-routed IP multicast, that source-routing header needs to include
the IP destination (multicast group) address. See e.g.: draft-eckert-msr6-rbs

> Arguably you can use the flow-label field for this so the header DOES NOT need to be larger. There are plenty of flow-label bits. But you are still going to have a control-plane, like BIER does. But maybe you can reuse that control-plane.

To get down to only the bits of the neighbors, the source-routing needs
to include these bits for every router on the distribution tree that you
want to send the packet on.  This is more than flow-label, but this is
exactly what RBS does.

> Its not clear at all to me, that operators would deploy this rather than just using an overlay, where when used properly there is no multicast state in the underlay.

One origin story of BIER is of coure exactly to provide improvements
over such solutions by service provider: Ingres-replication. And of course,
this is a matter of performance criteria which solution to pick. But
stateless source-routing that drives replication in the intervening hops
like BIER and all  derived idea do is still stateless in the core. And
it is similar in operations to steering as in SR (MPLS or SRv6). Aka:
its a lot closer in concept to unicast and thats always what in the
past has made multicast solutions more amenable to operators.

(but this is really not a novel argument for MSR6, but of course equally for BIER).

Cheer
   Toerless

> Dino
> 
> > On Oct 28, 2022, at 7:46 AM, Jeffrey (Zhaohui) Zhang <[e-mail]> wrote:
> > 
> > Hi,
> > 
> > Here is how I see it - I don't see one solution for all but separate solutions already exist or proposed with *IPv6-agnostic* data plane.
> > 
> > For "Stateless using MSR6 and still efficient in packet size when POSSIBLE" - BIERin6 or CGM2/RBS.
> > For "Stateful using MSR6 with proper control-plane when MUST" - P2MP tunnels (mLDP/RSVP/SR-P2MP)
> > 
> > I guess the crux is the following (the debate that we have a couple of years ago):
> > - "what is IPv6 native" - is it only if the BIER/RBS header is put into an IPv6 extension header?
> > - "must it be IPv6 native" - BIERin6, which is "transport" (L2 or tunnel or IPv4/IPv6) agnostic, just can't do the job?
> > 
> > Jeffrey
> > 
> > 
> > Juniper Business Use Only
> > 
> > -----Original Message-----
> > From: MBONED <[e-mail]> On Behalf Of Xiejingrong (Jingrong)
> > Sent: Friday, October 28, 2022 3:02 AM
> > To: Dino Farinacci <[e-mail]>; Toerless Eckert <[e-mail]>
> > Cc: BIER WG <[e-mail]>; [e-mail]; [e-mail]; [e-mail]
> > Subject: Re: [MBONED] [Msr6] MSR6 BOF 3rd Issue Category: More details are requested about the large scale use cases, including issue 8-11
> > 
> > [External Email. Be cautious of content]
> > 
> > 
> > Hi,
> > 
> > I got a time to read through this thread just now, and found the discussions very frank and very interesting.
> > If I understanding it correctly, Dino's point is that, efficient packet size is the highest priority in a multicast solution design choice, stateless and other "benefits" are all secondary.
> > Can we expect a solution that is:
> > Stateless using MSR6 and still efficient in packet size when POSSIBLE ?
> > Stateful using MSR6 with proper control-plane when MUST ?
> > 
> > Thanks
> > Jingrong
> > 
> > 本邮件及其附件可能含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
> > This e-mail and its attachments may contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
> > 
> > 
> > -----Original Message-----
> > From: Msr6 [mailto:[e-mail]] On Behalf Of Dino Farinacci
> > Sent: Tuesday, October 25, 2022 10:31 AM
> > To: Toerless Eckert <[e-mail]>
> > Cc: Yisong Liu <[e-mail]>; [e-mail]; [e-mail]; BIER WG <[e-mail]>; [e-mail]; Stig Venaas <[e-mail]>; [e-mail]
> > Subject: Re: [Msr6] MSR6 BOF 3rd Issue Category: More details are requested about the large scale use cases, including issue 8-11
> > 
> > Toerless, a packet without source routing gives more packet space to the user than one has source routing. Everything else is secondary and not relevant to this basic point.
> > 
> > Solve the problem, whatever you think it is, with a control-plane.
> > 
> > Dino
> > 
> >> On Oct 24, 2022, at 9:27 AM, Toerless Eckert <[e-mail]> wrote:
> >> 
> >> On Sun, Oct 23, 2022 at 07:54:09PM -0700, Dino Farinacci wrote:
> >>>> Think of a DC with 10,000 nodes and considering stateless multicast 
> >>>> source routing with 10,000 addressable destinations. 1280 min MTU 
> >>>> for ethernet is not a concern in such a network. Even if it
> >>> 
> >>> It is a concern if no user data is in the packet.
> >> 
> >> Rephrasing: Requiring a larger MTU than 1500 to support such an option 
> >> is not a limiting factor for such type of controlled network with 
> >> next-generation hardware (do you remember when we had to muck around 
> >> with 64kbyte jumbo packet support for HEPnet networks in the 1990th 
> >> and even in the early 200x - aka: networks with larger MTU have been around forever).
> >> 
> >>>> costs 50% or more overhead on e.g. a 1280 byte packet payload, 
> >>>> cutting
> >>> 
> >>> Its going to cost 1250 bytes for 10,000 destinations.
> >> 
> >> Right. And i provided the calculation how the overal amount of traffic 
> >> even with such a large header would be lower than replicating the 
> >> packet and sending it multiple times with smaller header to reach all destinations.
> >> 
> >> BIER for example itself is spec'ed to support up to 4096 bits, based 
> >> on wort case/largest deployment case considerations as of 10 years 
> >> ago. That might not be required for the current SP-WAN deployments, 
> >> but obviously, when we started BIER, we also looked into broader 
> >> use-case candidates than whats currently the BIER deployment focus.
> >> Thinking of another factor 2 as a possible maximum is not a big stretch of the imagination.
> >> 
> >>>> the addresing down to 5,000 and sending two packets across the 
> >>>> network would be more bandwidth incurred on it.
> >>> 
> >>> Sorry, your argument makes no sense.
> >> 
> >> Payload 1280. total number of egres routers: 10,000
> >> Sending one packet with 10,000 bit bitstring:      1250+1280 =  2530 byte
> >> Sending two packets with 5,000 bit bitstring:   2*(1250+625) =  3810 bytes
> >> 
> >>>> Splitting multicast across multiple packets also brings up the 
> >>>> unfairness concern of differential latency and the synchronization deciding "last-receiver"
> >>>> highest latency propagation latency.
> >>> 
> >>> But if you don't have to split it up across two packets, it is better for the user.
> >> 
> >> You are making my point. Of course i know how you do not want to, 
> >> because you are arguing for stateful multicast solutions at scale.
> >> 
> >>> You CANNOT argue this point. You might say wasting data packet bandwidth to elminate control state is a good tradeoff, but it clearly is not. And you won't be able to convince this point to anyone.
> >> 
> >> Customers are accepting the overhead of source routing headers over 
> >> managing forwarding state in networks. This applies both to unicast (SR vs. RSVP-TE) and multicast (BIER/MSR6).
> >> Customers have eliminated stateful solutions with e.g.: RSVP-TE in 
> >> favor of that. They have replaced stateful multicast with ingres replication to avoid state in the core.
> >> 
> >> These customers are looking at the overall traffic savings. The 
> >> reference is unicast, not stateful multicast. If i take a unicast 
> >> solution sending to 1000 to 10000 receivers it requires 1000x...10000x 
> >> of the payload size. If i give then a stateless multicast solution tat 
> >> requires 1x...3x of the payload size because of header, that IS preferrable over introducing a whole new stateful service into the network.
> >> 
> >> Larger MTUs btw. have ben common in many controlled networks forever.
> >> Remember all the requirements for HEPnet in the 1990th with networks 
> >> using up to 64k IP MTU ? Nowadays all those DC networks with RoCE also use larger MTU.
> >> 
> >> Of course, thinking of a header size of 1k is a stretch today, so most 
> >> of my colleagues also think that a 1/10th of this is a a reasonable 
> >> limit, but i think thats just too much grounded in backward fears and 
> >> not comparing the actual use-case benefits, especially simpliciy, 
> >> predictability, minimum latency/jitter. Those are going to be the criteria of interest going forward.
> >> 
> >>> If you want people to take msr6 seriously, you have to make good obvious tradeoffs.
> >> 
> >> But lets make sure we do not asume that tradeoffs for an SP-WAN based 
> >> on todays router hardware limits the ability for the best tradeoffs in a DC 5 years down the road.
> >> 
> >> [ I think we've seen this in the opposite direction with IPv6, where i 
> >> think everybody was happy to see min MTU raised to 1280 over IPv4, and 
> >> the Internet was happy to get the
> >> 64 bit routing address space (tongue in cheek ;-), except that 
> >> "everybody" didn't include all those IoT and other controlled 
> >> networks, that since then had to almost start an IETF area of their 
> >> own to come up with all those workaround to make IPv6 better fit those 
> >> networks (header compression, fragmentation, routing etc.). ]
> >> 
> >> I just don't want this to
> >> happen for MSR6, but i want MSR6 to be scaleable across a wider range 
> >> of networks, especially at the higher end in its core design. Thats 
> >> why i am happily being provocative here with the source-routing size 
> >> to have us think further. Especially when the IETF is mostly looking 
> >> (bcause of participartion) at mostly the SP-WAN market in the west, 
> >> which alas is not moving much, and ignoring that n countries like china there are still a lot more scale requirements to solve. WAN and DC.
> >> 
> >>>> And some key applictions in DCs may actually want to send 
> >>>> lowest-latency traffic to thousands of receivers.  Consider parallel 
> >>>> compute application worker management, like those customers have used since the early 200x in DC.
> >>>> Those packets may today need to go to thousands of parallel 
> >>>> instances and for fastest synchronization they should arrive at all 
> >>>> of them at the same, fastest time.
> >>> 
> >>> We can talk about low latency solutions once you give up the need to put so much state in a data packet. That is a different topic, and your data plane bloat won't solve either.
> >> 
> >> I respectfully disagree. Being able to send with an equal latency in 
> >> the usec range a packet to multiple destinations WITHOUT prior 
> >> estblishment of multicast state is one core benefit of stateless 
> >> multicast, and the data overhead of that is just a quantifyable cost that can easily be judged vs. the alternative (stateful) based on use-case.
> >> 
> >>> Note if a packet is delivered on a state based delivery tree, with no source-route, and you have a joiner downstream on the distribution tree that joins "at the same time as the packet is traveling down the tree", that new receiver can get the packet.
> >> 
> >> A high dynamic rate of join/leaves is a myth that we succumbed to when 
> >> designing multicast RMT solutions in the IETF. In fact it is the 
> >> stateless multicast with source routing that is the key enableer of 
> >> scaleable/rate-adaptive multicast transport solutions.  See 
> >> draft-ietf-bier-multicast-http-response
> >> (we'll have to rewrite this).
> >> 
> >>> With a source-route, any existing packets won't get delivered to that receiver. So you will have high join latency and missed opportunities to deliver packets already in flight to the new receiver.
> >> 
> >> This observation does not appply to applications where the sender 
> >> knows best who needs to get what. Which ultimately is the case in 
> >> almost all multicast applications. DASH/ABR over multicast (see above) 
> >> or distributed coordination via multicast.
> >> 
> >> Even without adaptive video, but most boring MPEG IPTV, the receiver 
> >> driven joins where a complete pain: Channel zapping where you really 
> >> don't want to receive duplicate traffic even for short periods 
> >> (limited receiver link BW), but you also don't want to join as soon as possible multicast, but get unicast until the next GOP.
> >> This switchover is done sooooo much easier with source-routing.
> >> 
> >> Even when it's not the case, join propagation times are in the order 
> >> of msec per hop, vs. typically much shorter packet forwarding times 
> >> host-to-host in networks up to metro size.
> >> 
> >>>> Of course, one would certainly like a stateless source routing 
> >>>> header design that does not require to carry the 10,000 bit receiver 
> >>>> information
> >>> 
> >>> "like"? How about it's a strong requirement for obvious reasons.
> >> 
> >> I should have said "unnecessary 10,000 bits". Akas: In the same way 
> >> that customers where happy to start SRv6 with 8*16=128 byte SRH 
> >> steering data, they also would like to see CRH. I was thinking of the same for bitstrings:
> >> If all i have is a 10,000 "flat" bitstring, customers can make the 
> >> easy calculation how this is e.g.: 2x..3x of the payload data in the 
> >> DC, so they'll go with it. But when its clear that we could compress 
> >> the header significantly when the number of receivers is more sparse, then they would of course want that (RBS).
> >> 
> >>>> if the addressed set is actually smaller (such as 200). And there 
> >>>> are proposals for that (dynamic source-route-header size based on size of receiver set).
> >>>> See e.g: draft-eckert-msr6-rbs.
> >>> 
> >>> So you will have multiple solutions for group size? That is a bad tradeoff too. And what happens when you go from 200 receivers 201, is there a major shift to a different solution?
> >> 
> >> [ There are working groups that had to claim the use-case was simple 
> >> and clear and a single solution easily feasible to get chartered. And 
> >> then they evolved into completely different use-cases and a wide range 
> >> of alternative solution component pieces, but never the original 
> >> use-case ;-) ]
> >> 
> >> What we have with MSR6 a different candidate proposals, and one of the 
> >> core part of the work of the proposed WG is to figure out what the 
> >> best compromise is for first WG and industry adopted mechanisms.
> >> 
> >> Obviosly, like Mr Fynman, i would like a unified source-routing-header 
> >> theory of the universe. To this end it is for example that we want to 
> >> investigate if the RBS option could also have favourable performnce 
> >> metrics over destination-only source-routing. aka: like BIER - MSR6 docs call this the "BE" mode (Best Effort).
> >> Even though RBS at its core is just a much better evolution for tree 
> >> engineering over BIER-TE.
> >> 
> >> If this fully unified option is insufficient, we most likely would 
> >> arrive at one option for BE and one for tree engineering (TE). Aka:
> >> maybe RBS for TE and flat-bitstring for BE, and beside all the IPv6 
> >> forwarding plane specifics, we might just increase the maximum 
> >> supported header size for both based on hardware capability in future 
> >> routers
> >> (aka: today we have 512 byte examinable packet heder in routers, then 
> >> it likely could be at least 1024). But thats personal conjecture.
> >> 
> >>> That sounds far worse than what we experirenced switching from shared-tree to source-tree.
> >> 
> >> I may have worked with more customers having pain with that across 
> >> more different HW accelerated platforms than you did, so my mileage 
> >> may vary. But i have a hard time thinking these two technology aspects could be compared fairly in the same ballpark.
> >> 
> >> IMHO, packet header size is an easily quantified cost vs. benefit 
> >> evaluation for the customers.  Operations of RTP/SPT switchover is an 
> >> ugly technology detail nobody should need to understand, but if you're 
> >> operating a PIM-SM network you MUST understand all it's bloody 
> >> intricacies. And then there are the customers threatening you with a lot of money and explaining you that their definition of IP datagram forwarding is:
> >> no loss, no jitter, no reorder. And you know how you can build 
> >> multicast to do it, and its actually a nice way to justify the high cost of your product. Just RPT/SPT is not the way.
> >> (but source-routing is one such option ;-).
> >> 
> >>>> Wrt to the receiver tracking: Remember that in end-to-end 
> >>>> applications only the sender may need to be involved in calculating 
> >>>> the receivers, (No network control plane harmed!).
> >>> 
> >>> Then you have the same problem with head-end replication at the source, as we do today with CDNs. You are just moving the problem and not solving the problem where a source just sends packets to a group address.
> >> 
> >> The source is the one sending the source-routed multicast packet
> >> 
> >>> Today, multicast sending from a source CANNOT get more efficient. You just send one UDP packet to a multicast group. That is pretty simple in my mind and can't get any simpler. So anything you change will add overhead and of course a non-starter.
> >> 
> >> We did outsource the source discovery from network layer to 
> >> application land when we introduced SSM, because the network as doing 
> >> a shitty job at scaling for that (RPT/SPT just being one part of the problem).
> >> 
> >> We're moving membership management out of the network with 
> >> source-routing. Whether its done locally in the source aplication or 
> >> with a help of a controller, it is so much easier to NOT do it in the 
> >> network where routers always have constrained contrl plane, and not do 
> >> it distributed in 20 hops along a path, but only in source or 
> >> controller, both of which can much more easily have arbitrary amount 
> >> of CPU compared to network devices - and a source especially only 
> >> needs to bother about its own traffic.
> >> 
> >> lets also remind the rest of the audience here that i am hopefully 
> >> fair to say that this critique of yours is not specific to MSR6 but also applies to BIER.
> >> 
> >>>> Those host nodes typically can do a shi.load of compute in 
> >>>> high-speed CPU cache when they are DC servers.  In the mentioned 
> >>>> parallel compute worker management, this would for example be a 
> >>>> dynamic subset calculation of 10,000 parallel workers based on ongoing performance telemetry.
> >>>> I bet those existing large-scale distributed compute apps already 
> >>>> have to spend orders of magnitude more compute than converting any 
> >>>> such subset into a bitstring.
> >>> 
> >>> They can do better and faster by not doing this.
> >> 
> >> First of all, this is not true for native source-routing apps like i 
> >> mentioned above, where the application sender can now perfectly manage 
> >> things like sending excactly only whats needed during channel zapping 
> >> of a receiver (receiver in old TV channel source-routing header, 
> >> followed by unicasted cached unicast reference frames of new TV 
> >> channel, followed by receiver in new TV channel source-routing header). Likewise all the similar application examples for other applications in DC.
> >> 
> >> Btw: It's one of the big pains that we never so far got into writing 
> >> down more elaborately all those application benefits of 
> >> source-routing. Alas, this is because no application developer can buy today really routers with BIER to experiment with.
> >> 
> >> Secondly: Even when we have "good-old-ip-multicast" applications with 
> >> receiver joins, the overall solution complexity network+application 
> >> goes down by moving to SSM, and it even further moves down when we do 
> >> the receiver tracking in the SSM sender.
> >> 
> >> Cheers
> >>   Toerless
> >> 
> >>> Dino
> >>> 
> >> 
> >> --
> >> ---
> >> [e-mail]
> > 
> > --
> > Msr6 mailing list
> > [e-mail]
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/msr6__;!!NEt6yMaO-gk!DBCC8DrBuiDv8c751ocIPw4VAIzivgMtzOeKRdDM6ctgWWLLJ-Otyii9_3v8ShIsvM8lDDCzwJbpGkkDQZU9mq9V2Wahx7UB$
> > 
> > _______________________________________________
> > MBONED mailing list
> > [e-mail]
> > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/mboned__;!!NEt6yMaO-gk!DBCC8DrBuiDv8c751ocIPw4VAIzivgMtzOeKRdDM6ctgWWLLJ-Otyii9_3v8ShIsvM8lDDCzwJbpGkkDQZU9mq9V2cO0DVYj$

-- 
---
[e-mail]