Re: [Coin] Fwd: The Future of P4, Revisited

[Toerless Eckert <[e-mail]>]

On Fri, May 19, 2023 at 12:16:20PM -0700, Hesham ElBakoury wrote:
> Broadcom folks have not tried to run NPL programs on Tofino. If someone
> tried doing this please let us know.

Sorry if anything i wrote was confusing to that end. I was just browsing what
whiteboxes are available on the market, and didn't mean to imply what you said,
and only Broadcom whiteboxes had the type of TM i was interested, e.g.: TSN,
which could be repurposed for IETF DetNet. No Tofino box doing the same,
unless one wanted to reimplement TSN with Intel SmartNIC.

But if what you wrote was not meant as a direct reply to my email you replied to:

Early in the P4 life cycle, several other vendors tried to jump on the P4
bandwaggon, announcing P4 support for their chips. I think little of that
did pan out, and i am not aware that anything of this actually survived,
except of course options for CPU forwarding (RPI P4, Hemants compiler) and
likely SmartNICs that wouldn't qualify as 'CPU'.

Sure, there are FPGA P4 option such as on Xilinx, but i have yet to find
someone who was not frustrated with instability/problems of that option.
That btw. was my whole starting point of Tofino not going forward - there
is no good alternative for high-speed, low-cost ASIC with P4.

Cheers
    Toerless

> Hesham
> 
> On Fri, May 19, 2023, 11:23 AM Toerless Eckert <[e-mail]> wrote:
> 
> > I think today there simply is no real programmable TM in hardware.
> >
> > Instead, every hardware has some configurable queuing/scheduling and
> > AQM, and the programmable stage at best can program the classical
> > parameters such as queue and traffic class within a queue.
> >
> > The best high-speed PoC hardware today towards programmable TM are likely
> > the few whitebox with Tofino and some form of Intel SmartNIC. There are
> > also Broadcom switches with FPGA, which would be even better, if not for
> > the little access to NPL for the broadcom ASIC (i still have to evaluate
> > how real access to NPLS is for non-vendors).
> >
> > Cheers
> >     Toerless
> >
> > On Fri, May 19, 2023 at 11:18:01AM -0700, Hesham ElBakoury wrote:
> > > It is worth noting that Broadcom NPL does not provide Traffic Management
> > > support.
> > >
> > > Hesham
> > >
> > > On Fri, May 19, 2023, 11:11 AM Toerless Eckert <[e-mail]> wrote:
> > >
> > > > On Fri, May 19, 2023 at 01:37:05PM -0400, [e-mail] wrote:
> > > > > P4 supports variable length parsing.
> > > >
> > > > Not via the parser itself, but through explicit P4 programming, which
> > is
> > > > a significant limitation. I can find our older papers on variable
> > length
> > > > packet parsing issues. Deparsing is then also equally more complex.
> > > >
> > > > P4 is based on a well known type of parsing hardware, which is employed
> > > > in Tofino. But there are more intelligent parsing engines.
> > > >
> > > > > Regarding memory optimizations on cpu, vpp has optimized memory for
> > both
> > > > instructions and data.
> > > >
> > > > For processing that needs to operate on more than just integers,
> > > > such as monitoring, accounting and inference, you definitely want
> > > > a specification language that supports the desired data types natively.
> > > > Such as real or fixed-point numbers.  Even allow to define new types.
> > > > But also beyond that.  For example, in one of my drafts, i need to do
> > > > have bit-accurate pointers for calculation into complex bitstring
> > > > structures. Same is likely true if we where to pseudocode
> > > > specify some of the IoT header compression mechanisms, or other
> > > > compressed structures.
> > > >
> > > > Production VPP code is certainly not useful as specification reference
> > > > code.
> > > > I know how much of that code was written by developers (Ed et. al.)
> > > > who knew exactly what x86 instructions where generated from C code and
> > > > their cycle count. Which of course was (and i guess still is) then a
> > good
> > > > amount of re-optimization need for ARM (or other CPU). Not to talk
> > about
> > > > SIMD.
> > > >
> > > > > All these optimizations are available in my company's P4toVPP
> > compiler.
> > > >
> > > > Sure, but a better specification language than P4 would be even more
> > > > opportunity for
> > > > a future versions of a compiler like yours.  For example, fixedpoint
> > math
> > > > looks ugly in P4. Good enoough for validation code, but not for a spec
> > > > language.
> > > > It could look very nice if the language had such data
> > > > types natively. And your compiler could translate into a fitting P4
> > > > fixedpoint math library. Or extern libraries, such as for actual
> > floating
> > > > point calculation. Or even skip mapping into P4 completely and creating
> > > > better code directly mapping to the underlying platform features.
> > > >
> > > > > TM is still an open issue to work on.
> > > >
> > > > There is good work happening for high-speed PIFO type FPGA forwarding
> > code.
> > > > I think some form of extended PIFO functionality could become the
> > basis for
> > > > programmable TM in the future. Still has to be seen how well it would
> > > > possible
> > > > to efficiently PoC this on lower end CPU forwarders. If this direction
> > pans
> > > > out, then the forwarding plane impact is just the compute of the TM
> > > > parameters such as rank/timestamps - and processing on dequeueing.
> > > >
> > > > Cheers
> > > >     Toerless
> > > >
> > > > >
> > > > > Hemant
> > > > >
> > > > > -----Original Message-----
> > > > > From: 'Toerless Eckert' <[e-mail]>
> > > > > Sent: Friday, May 19, 2023 12:33 PM
> > > > > To: Haoyu Song <[e-mail]>
> > > > > Cc: [e-mail]; 'Hesham ElBakoury' <[e-mail]>;
> > > > [e-mail]; 'Bernier, Daniel' <[e-mail]>;
> > > > 'Marie-Jose Montpetit' <[e-mail]>; 'coin' <[e-mail]
> > >;
> > > > 'coinrg-chairs' <[e-mail]>
> > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > >
> > > > > Right.
> > > > >
> > > > > For high speed router/switch (>= 100gbps/interface) validation:
> > > > >   P4 implementations on Tofino are great as proof of concept - if it
> > can
> > > > be done.
> > > > >   If P4 implementation on Tofino does not work, other proof points
> > are
> > > > needed,
> > > > >
> > > > > For medium speed "software forwarding" validation:
> > > > >   P4 creates too many limitations. All CPU have float, and/or SMDI,
> > > > and/or
> > > > >   more flexible memory access options and likely several other
> > aspects
> > > > not
> > > > >   easily utilized by P4.
> > > > >
> > > > > For generic specification:
> > > > >   IMHO above limitations are at the core of why P4 is insufficient.
> > > > >   I very much like several structural aspects of P4, they would be
> > great
> > > > >   starting points for general purpose spec. Such as header parsing
> > specs
> > > > >   (need to be extended for variable length though).
> > > > >   Aka: Anything we can use from P4 which is more/better declarative
> > than
> > > > >   C/C++ is IMHO useful. Just lets not introduce constraints:
> > Validation
> > > > on
> > > > >   different type of forwarding platforms is different from a single
> > > > >   specification.
> > > > >   And we are also missing good DSL spec options for TM i fear...
> > > > >
> > > > > Cheers
> > > > >     Toerless
> > > > >
> > > > > On Fri, May 19, 2023 at 04:02:57PM +0000, Haoyu Song wrote:
> > > > > > I think the discussion is off the point. We are discussing whether
> > or
> > > > not P4 is suitable for a standard specification language for general
> > > > dataplane processing and forwarding applications. To fulfill that
> > purpose,
> > > > it needs to be target/architecture independent; it needs to be concise
> > and
> > > > expressive to cover various dataplane functions which may not be
> > > > conceivable today (e.g., in-network computing applications).
> > > > > > Unfortunately, I don't think P4 can meet these requirements now
> > > > > > (imagine all the pseudo code in RFCs are translated to P4). Of
> > course,
> > > > > > it can be limited to just support certain functions, e.g., to
> > document
> > > > > > the header formats and parsing process (but here I don't see the
> > > > > > advantage of P4 either)
> > > > > >
> > > > > > Haoyu
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: [e-mail] <[e-mail]>
> > > > > > Sent: Thursday, May 18, 2023 6:19 PM
> > > > > > To: 'Toerless Eckert' <[e-mail]>
> > > > > > Cc: Haoyu Song <[e-mail]>; 'Hesham ElBakoury'
> > > > > > <[e-mail]>; [e-mail]; 'Bernier, Daniel'
> > > > > > <[e-mail]>; 'Marie-Jose Montpetit'
> > > > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > > > <[e-mail]>
> > > > > > Subject: RE: [Coin] Fwd: The Future of P4, Revisited
> > > > > >
> > > > > > Toerless,
> > > > > >
> > > > > > Only externs defined in core.p4 are in a library. Externs defined
> > in
> > > > an architecture include file, e.g., v1model.p4 are implemented in
> > > > target-specific back end and are extensions. An example of a target
> > extern
> > > > is vpp accelerator in Octeon 10. Neither P4-16 spec nor PNA support
> > > > floating point and neither does my compiler. Floating point in used be
> > > > AI/ML. The float point numbers would have to be converted to integer.
> > > > Google's Tensor chip also converts float to integer for ML.
> > > > > >
> > > > > > If you want to work with the linux kernel, see the ebpf
> > architecture
> > > > include file,
> > > > https://github.com/p4lang/p4c/blob/main/p4include/ebpf_model.p4.
> > Doesn't
> > > > the file have externs which have totally abstracted linux kernel data
> > > > structs and functions?
> > > > > >
> > > > > > Hemant
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Coin <[e-mail]> On Behalf Of 'Toerless Eckert'
> > > > > > Sent: Thursday, May 18, 2023 8:23 PM
> > > > > > To: [e-mail]
> > > > > > Cc: 'Haoyu Song' <[e-mail]>; 'Hesham ElBakoury'
> > > > > > <[e-mail]>; [e-mail]; 'Bernier, Daniel'
> > > > > > <[e-mail]>; 'Marie-Jose Montpetit'
> > > > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > > > <[e-mail]>
> > > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > > >
> > > > > > Hemant,
> > > > > >
> > > > > > Externs and functions are not an extension to the language.
> > > > > > They are really just libraries.
> > > > > >
> > > > > > Syntactic changes/extensions such as for parsing/deparsing, new
> > types,
> > > > such as float/double - those would be language extensions.
> > > > > >
> > > > > > This makes the use of P4 cumbersome on any platform that can do
> > more
> > > > than the worst-case-chip against which the P4 language is designed
> > (aka:
> > > > Tofino).
> > > > > >
> > > > > > For example on your OCTEON P4 compiler: The ARM A72 does support
> > > > floating point numbers. If you implemented support for them via the
> > > > mechanisms supported by P4, the way i understood it, you could still
> > not
> > > > write simple P4 programs for OCTEON then like:
> > > > > >
> > > > > >     float a, b, c
> > > > > >     a = b * c / 3.2;
> > > > > >
> > > > > > You may have a set of externs/functions through which you could do
> > > > this, but it would look a lot more ugly.
> > > > > >
> > > > > > Right ? If not, pls. explain - i would be happy!
> > > > > >
> > > > > > And Haoyu's argument was that when we want to suggest to use P4 as
> > the
> > > > reference specification or base forwarding programming in the linux
> > kernel,
> > > > then this need to hide stuff outside of P4 written externs and
> > functions
> > > > does limit P4's useful as a complete specification and reference code
> > > > language.
> > > > > >
> > > > > > Cheers
> > > > > >     Toerless
> > > > > >
> > > > > > On Thu, May 18, 2023 at 04:56:31PM -0400, [e-mail] wrote:
> > > > > > > Why do you care for internal behavior of checksum when the P4
> > api is
> > > > able to provide data to compute checksum.
> > > > > > >
> > > > > > >
> > https://github.com/p4lang/p4c/blob/main/p4include/v1model.p4#L505
> > > > > > >
> > > > > > > Hemant
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Coin <[e-mail]> On Behalf Of Haoyu Song
> > > > > > > Sent: Thursday, May 18, 2023 3:30 PM
> > > > > > > To: 'Toerless Eckert' <[e-mail]>; [e-mail]
> > > > > > > Cc: 'Hesham ElBakoury' <[e-mail]>;
> > [e-mail];
> > > > > > > 'Bernier, Daniel' <[e-mail]>; 'Marie-Jose
> > Montpetit'
> > > > > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > > > > <[e-mail]>
> > > > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > > > >
> > > > > > > Toerless,
> > > > > > >
> > > > > > > This is exactly the problem: you can't use the core language to
> > > > describe an arbitrary dataplane function.
> > > > > > > "Extern" can be a blackbox but what need is the spec for what's
> > in
> > > > the blackbox.
> > > > > > >
> > > > > > > For P4 spec: " Extern objects are architecture-specific
> > constructs
> > > > that can be manipulated by P4 programs through well-defined APIs, but
> > whose
> > > > internal behavior is hard-wired (e.g., checksum units) and hence not
> > > > programmable using P4"
> > > > > > >
> > > > > > > Haoyu
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: 'Toerless Eckert' <[e-mail]>
> > > > > > > Sent: Thursday, May 18, 2023 12:15 PM
> > > > > > > To: [e-mail]
> > > > > > > Cc: Haoyu Song <[e-mail]>; 'Hesham ElBakoury'
> > > > > > > <[e-mail]>; [e-mail]; 'Bernier, Daniel'
> > > > > > > <[e-mail]>; 'Marie-Jose Montpetit'
> > > > > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > > > > <[e-mail]>
> > > > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > > > >
> > > > > > > On Thu, May 18, 2023 at 02:19:12PM -0400, [e-mail]
> > wrote:
> > > > > > > > A specific P4 architecture, e.g., PNA, may override the base P4
> > > > spec. PNA supports writing table entry by data plane. Using registers
> > in P4
> > > > is a bit clunky but state can be maintained. Anyone is welcome to
> > present a
> > > > better solution to the P4 forums.
> > > > > > >
> > > > > > > So far, my limited experience has been, that the language itself
> > has
> > > > rather evolved over its series of spec to further reduce the base
> > language
> > > > functionality and move any improvemenets into implementation extensions
> > > > ("extern") calls, so that no possible hardware would not be able to not
> > > > support any base language feature.
> > > > > > >
> > > > > > > This is a good strategy to gain more acceptance in the industry,
> > but
> > > > > > > not to get to the easiest to use forwarding plane behavioral
> > > > > > > definition language
> > > > > > >
> > > > > > > For exmple, i would need axtual syntactical language extensions
> > to
> > > > define a good way to parse andd deparsing variable length header
> > fields.
> > > > This can not just be done with extern calls.
> > > > > > >
> > > > > > > Has P4 adopted the approach to have optional language feature
> > > > extensions ?
> > > > > > > Sorry. Did not have time to follow the spec evolution in detail.
> > > > > > >
> > > > > > > Cheers
> > > > > > >     Toerless
> > > > > > > > Hemant
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Haoyu Song <[e-mail]>
> > > > > > > > Sent: Thursday, May 18, 2023 1:56 PM
> > > > > > > > To: Hesham ElBakoury <[e-mail]>
> > > > > > > > Cc: [e-mail]; Toerless Eckert <[e-mail]>;
> > > > > > > > hemant=[e-mail] <[e-mail]>; Bernier,
> > > > > > > > Daniel <[e-mail]>; Marie-Jose Montpetit
> > > > > > > > <[e-mail]>; coin <[e-mail]>; coinrg-chairs
> > > > > > > > <[e-mail]>
> > > > > > > > Subject: RE: [Coin] Fwd: The Future of P4, Revisited
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi Hesham,
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Please see P4 spec section 6.5.2 for the current “stateful”
> > > > support.
> > > > > > > > So far the “table” element, which is the most important
> > construct
> > > > > > > > for a P4 program,  is not stateful (i.e., dataplane writable)
> > > > > > > >
> > > > > > > > I think it’s a direction to extend P4 for better stateful
> > support
> > > > (part of my recent research). At least now it’s unnatural and
> > difficult to
> > > > describe many stateful functions.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Haoyu
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Coin <[e-mail] <mailto:
> > [e-mail]>
> > > > >
> > > > > > > > On Behalf Of Hesham ElBakoury
> > > > > > > > Sent: Thursday, May 18, 2023 10:42 AM
> > > > > > > > To: Haoyu Song <[e-mail]
> > > > > > > > <mailto:[e-mail]> >
> > > > > > > > Cc: [e-mail] <mailto:[e-mail]> ;
> > Toerless
> > > > > > > > Eckert <[e-mail] <mailto:[e-mail]> >;
> > > > > > > > hemant=[e-mail]
> > > > > > > > <mailto:hemant=[e-mail]> ; Bernier, Daniel
> > > > > > > > <[e-mail] <mailto:[e-mail]> >;
> > > > > > > > Marie-Jose Montpetit <[e-mail]
> > > > > > > > <mailto:[e-mail]> >; coin <[e-mail]
> > > > > > > > <mailto:[e-mail]> >; coinrg-chairs <
> > [e-mail]
> > > > > > > > <mailto:[e-mail]> >
> > > > > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi Haoyu,
> > > > > > > >
> > > > > > > > Actually, P4 supports dataplane-modifiable tables -- see PNA.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > More generally, the *language* is fully extensible. You can
> > have
> > > > whatever architecture and state externs you want. So one needs to be
> > > > careful to separate language (non) limitations from target limitations.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hesham
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, May 18, 2023, 10:15 AM Haoyu Song <
> > > > [e-mail] <mailto:[e-mail]> > wrote:
> > > > > > > >
> > > > > > > > Hi Hesham,
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I said “it’s very limited”. P4 table is only readable (i.e.,
> > not
> > > > writable) by dataplane, so it basically eliminates any dataplane
> > stateful
> > > > function that need to use tables. The stateful function can only use
> > > > register arrays but unfortunately registers are local to a pipeline
> > stage
> > > > so the state update logic must be very simple and can be finish in a
> > single
> > > > stage. What you said “event” must be something very simple. Use case
> > such
> > > > as stateful load balancer can’t be implemented by P4.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > There are some works addressing the issue. FlowBlaze is the
> > most
> > > > recent one, which needs a new chip architecture but it is still
> > limited to
> > > > simple stateful functions which can be realized in a single pipeline
> > stage.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Haoyu
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Hesham ElBakoury <[e-mail]
> > > > > > > > <mailto:[e-mail]> >
> > > > > > > > Sent: Thursday, May 18, 2023 10:00 AM
> > > > > > > > To: Haoyu Song <[e-mail]
> > > > > > > > <mailto:[e-mail]> >
> > > > > > > > Cc: [e-mail] <mailto:[e-mail]> ;
> > Toerless
> > > > > > > > Eckert <[e-mail] <mailto:[e-mail]> >;
> > > > > > > > hemant=[e-mail]
> > > > > > > > <mailto:[e-mail]>
> > > > > > > > ; Bernier, Daniel <[e-mail]
> > > > > > > > <mailto:[e-mail]> >; Marie-Jose Montpetit
> > > > > > > > <[e-mail] <mailto:[e-mail]> >; coin
> > > > > > > > <[e-mail] <mailto:[e-mail]> >; coinrg-chairs
> > > > > > > > <[e-mail] <mailto:[e-mail]> >
> > > > > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi Haoyu,
> > > > > > > >
> > > > > > > > You say: "P4 has very limited support for stateful processing"
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I am not sure I can agree with your characterization of P4.
> > > > Perhaps can you elaborate on this.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > Hesham
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, May 18, 2023, 9:19 AM Haoyu Song <
> > [e-mail]
> > > > <mailto:[e-mail]> > wrote:
> > > > > > > >
> > > > > > > > Interesting discussion. See my comments below  [HS]
> > > > > > > >
> > > > > > > > Haoyu
> > > > > > > >
> > > > > > > > > For example, in the multicast drafts i write, we use
> > > > > > > > > C-pseudocode to specify behavior, but we do attempt to
> > implemnt
> > > > on Tofino in P4.
> > > > > > > > > Should we really use P4 code for the RFC specs... ? (much
> > longer
> > > > > > > > > than C Pseudocode). Aka: quite selfish (but IETF relevant ;-)
> > > > reason to highlight this point.
> > > > > > > >
> > > > > > > > [EH]: This is an area I'm very interested in. Having a
> > > > standardized and formal language to describe protocols and behavior can
> > > > bring a lot of functionality and benefits to the IETF.
> > > > > > > > My initial thinking is that having such a blueprint, the IETF
> > > > could generate tools to create a reference implementation that can be
> > used
> > > > for interoperability purposes therefore decreasing time to test and
> > > > implement protocols and therefore RFC publications.
> > > > > > > >
> > > > > > > > [HS] P4 can only describe dataplane behaviors, so any control
> > > > plane stuff is out of scope. For dataplane, if it's used to describe
> > header
> > > > format, it's not better than the "struct" in C. The language uses the
> > > > match-action table abstraction with an implication of pipeline
> > > > implementation which may make it cumbersome or even impossible to
> > describe
> > > > the  behavior (e.g.,  P4 has very limited support for stateful
> > processing).
> > > > In general, I don't think P4 at its current form can undertake the
> > role for
> > > > formal protocol specification.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > ---
> > > > > > > [e-mail]
> > > > > > > --
> > > > > > > Coin mailing list
> > > > > > > [e-mail]
> > > > > > > https://www.irtf.org/mailman/listinfo/coin
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > ---
> > > > > > [e-mail]
> > > > > >
> > > > > > --
> > > > > > Coin mailing list
> > > > > > [e-mail]
> > > > > > https://www.irtf.org/mailman/listinfo/coin
> > > > > > --
> > > > > > Coin mailing list
> > > > > > [e-mail]
> > > > > > https://www.irtf.org/mailman/listinfo/coin
> > > > >
> > > > > --
> > > > > ---
> > > > > [e-mail]
> > > >
> > > >
> > > >
> > > > --
> > > > ---
> > > > [e-mail]
> > > >
> >
> > --
> > ---
> > [e-mail]
> >

-- 
---
[e-mail]