Re: [Bier] ASIC restrictions

[Toerless Eckert <[e-mail]>]

Thanks, Vasili, Tony

My experience has been that you need to throw really big money opportunities into
the front of product marketing, which then goes to chip development (or OEM) and
requests better hardware.

I lived through that when platforms/chip-designers complained bitterly about
moving from IPv4 to IPv6 with TCAM or MTRIE lookups, and of course, when i wanted
to have IPv6 (S,G) lookups (256 bits!) they complained bitterly again. Until
luckily someone asked for IPv6 5 tuple (unicast) ACLs, which required even longer lookups
and in result enabled (S,G) multicast lookups as well.

And i am not going to repeat all the details of the challenges that we as an
industry did have with High Energy Physics jumbo packets and the like. Several similar
stories..

So, it is definitely very prudent to be able to show value of any new system with
the currently feasible lookups such as 256 bit (which is why we concentrated out
initial simulation on that), but also have the system be able
to scale up to longer lookups, so big customers/use-cases can demand support in
future generations. Aka: i would not want to bet on such a strict "longer"
lookup requirement as IPv6 had it when IPv4 was standard. But once there are use-cases
thst make money and they go into a hardwre churn phase, there are always opportunities
to get more/longer lookups if shown beneficial.

Wrt SRH: I still would like to write a draft explaining how i could see an
SRH equivalent stateless multicast header, and we can discuss the length aspect
then. In general, i wonder, when we are talking in SPRING about compressing
the steering header elements, why we could not also consider compressing the base
header. The IoT community has done a lot more comprssion of IPv6 (and transport)
over the years, but they can throw a lot of per-packet CPU at the problem,
whereas in our case, we may just be happy with all of IPv6 except for shorter
addresses for a limited domain. Just saying.

Wrt recirculation: Whether or not that is actually something you want to avoid
very much depend on the chip design. It does seem to be undesirably expensive
on Tofino, but for well built chips, where you have some N packets throughput,
you design forwarding such that you can input some e.g.: N * 1.1, and any recirculated
packet just takes away from that input bandwidth. For sustained throughput,
you would not even need 1.1, but just 1.0 because ultimately, you can not send out
more than N packets anyhow,  whether you receive and send N unicast packets, or
you receive N/100 multicast packets, each of which is replicated 100 times. You
still do N packet (header) processing, without or with recirculation. 1.1 is only
needed to allow during recirculation of packets to simultaneously process enough
arriving (unicast) packets so that there is not (too much) head of line blocking.

It is quite common though to have replication through lookup of multicast
tables, such as presented in Steffen/Michaels P4 presentation, but i am not
too sure, how much post-processing is then still possible. One has to remember that
every BIER packet copy can have a different bitstring, because of the bit clear
operation which is specific to each copy. Usually, the only post-processing
on platforms with multicst table lookup is then limited to some L2 adjacencies.
If Tofino can actually still do a per-packet rewrite of the bitstring (as required by BIER)
after such a replication, then that would speak for Tofino. If Tofino can not
do that, then that would speak for BIER-TE, because BIER-TE does
only require an ingres clearing of bits, common for all packet copies (except for
one crazy optional feature). 

Cheers
    Toerless

On Fri, Nov 11, 2022 at 11:32:28AM +0000, Tony Przygienda wrote:
> yes, discussion has been many times but in 2 quips
> 
> 1. high end chips at least are not really limited by the overall size but
> the size they can put on the "really, really fast packet header
> processing scratchboard". And then you recirculate if you blow out that
> size. Which halves the throughput and hence the problem. That's, as you
> have seen in e.g. P4 work that was presented, makes things that work fine
> in smaller labs, networks, not feasible in large deployments in terms of
> economics. Making the scratchboard wider is very expensive in terms of
> die/power etc/etc obviously so it needs really strong economic drivers.
> 2. BIER introduced a way to deal with the problem via the concept of sets
> (and subdomains) and that's about the best engineering tradeoff on high end
> chips that's viable. ON good chips replicating a packet three times instead
> of twice is till much cheaper than recirculating from engineering
> perspective. In very loose terms explanation lies in the fact that holding
> stuff on-chip is very expensive, pushing it out the box is significantly
> cheaper (in a sense timexdelay buffer is way cheaper than on-die buffer)
> 
> BIER has been built as architecture by folks that were fighting those
> silicon things from the early days of the IP technology and many of them
> _way_ before ;-)
> 
> --- tony
> 
> On Thu, Nov 10, 2022 at 6:40 PM Vasilenko Eduard <vasilenko.eduard=
> [e-mail]> wrote:
> 
> > Dear multicast experts,
> >
> > I am not subscribed to this alias, I am not even a multicast expert, I am
> > a stranger here.
> >
> > When I heard today the comment that 1280B+1280B is the challenge from
> > MTU's point of view – I reacted that it is not the biggest problem, a much
> > bigger problem would be the overall size of all headers that particular
> > Chip could process (no way for 1280Bytes, never!).
> >
> > Toerless Eckert asked me to put my comment here.
> >
> > Well, I did believe that it is well-known and some sort of obvious.
> >
> > Some routing switches (for DC/Cloud) still have the restriction 128Bytes
> > for all headers (including MAC, VLAN, GTP, SRv6, etc).
> >
> > Some high-high-end Telco routers are capable to process 384bytes – it is
> > probably the upper limit now (again, for all L2-L4 headers).
> >
> > When I have seen the first BIER IETF presentation in 2016 - it immediately
> > comes to my mind that “not all vendors would be capable to implement this”.
> >
> > It would be not polite to say at least 2 names here – I could not respond
> > for other vendors.
> >
> > I have seen 1k routers network (and many in the range between 256 and 512
> > PEs), but 1k is already 128bytes just for BIER bit-field.
> >
> > Especially problem would be in the combination with SRv6 which could have
> > an SRH header up to 208bytes.
> >
> > Of course, it is possible to break E2E, split for Areas/Domains, and
> > stitch by Gateways.
> >
> > And you discussed today the way how to localize bit patterns – it is
> > probably some sort of automatic split for Areas (I did not read the draft).
> >
> > I did always believe that “headers size” is the BIER's primary problem.
> > Hence, it was probably discussed here many-many times.
> >
> > Sorry, if I stepped on something well discussed again. As I said: I am a
> > stranger. I am just passing by. Sorry, for the point if it is a triple
> > duplicate.
> >
> >
> >
> > Your architecture is so nice (stateless) – I like it very much. And I am
> > sure that my employer has no problem with header sizes. BIER foreverJ
> >
> >
> >
> > [image: cid:image001.png@01D3A7DF.E7D86320]
> >
> > Best Regards
> >
> > Eduard Vasilenko
> >
> > Senior Architect
> >
> > Europe Standardization & Industry Development Department
> >
> > Tel: [phone]
> >
> >
> > _______________________________________________
> > BIER mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/bier
> >



-- 
---
[e-mail]