Re: [Coin] Fwd: The Future of P4, Revisited

[Toerless Eckert <[e-mail]>]

I think today there simply is no real programmable TM in hardware.

Instead, every hardware has some configurable queuing/scheduling and
AQM, and the programmable stage at best can program the classical
parameters such as queue and traffic class within a queue. 

The best high-speed PoC hardware today towards programmable TM are likely
the few whitebox with Tofino and some form of Intel SmartNIC. There are
also Broadcom switches with FPGA, which would be even better, if not for
the little access to NPL for the broadcom ASIC (i still have to evaluate
how real access to NPLS is for non-vendors).

Cheers
    Toerless

On Fri, May 19, 2023 at 11:18:01AM -0700, Hesham ElBakoury wrote:
> It is worth noting that Broadcom NPL does not provide Traffic Management
> support.
> 
> Hesham
> 
> On Fri, May 19, 2023, 11:11 AM Toerless Eckert <[e-mail]> wrote:
> 
> > On Fri, May 19, 2023 at 01:37:05PM -0400, [e-mail] wrote:
> > > P4 supports variable length parsing.
> >
> > Not via the parser itself, but through explicit P4 programming, which is
> > a significant limitation. I can find our older papers on variable length
> > packet parsing issues. Deparsing is then also equally more complex.
> >
> > P4 is based on a well known type of parsing hardware, which is employed
> > in Tofino. But there are more intelligent parsing engines.
> >
> > > Regarding memory optimizations on cpu, vpp has optimized memory for both
> > instructions and data.
> >
> > For processing that needs to operate on more than just integers,
> > such as monitoring, accounting and inference, you definitely want
> > a specification language that supports the desired data types natively.
> > Such as real or fixed-point numbers.  Even allow to define new types.
> > But also beyond that.  For example, in one of my drafts, i need to do
> > have bit-accurate pointers for calculation into complex bitstring
> > structures. Same is likely true if we where to pseudocode
> > specify some of the IoT header compression mechanisms, or other
> > compressed structures.
> >
> > Production VPP code is certainly not useful as specification reference
> > code.
> > I know how much of that code was written by developers (Ed et. al.)
> > who knew exactly what x86 instructions where generated from C code and
> > their cycle count. Which of course was (and i guess still is) then a good
> > amount of re-optimization need for ARM (or other CPU). Not to talk about
> > SIMD.
> >
> > > All these optimizations are available in my company's P4toVPP compiler.
> >
> > Sure, but a better specification language than P4 would be even more
> > opportunity for
> > a future versions of a compiler like yours.  For example, fixedpoint math
> > looks ugly in P4. Good enoough for validation code, but not for a spec
> > language.
> > It could look very nice if the language had such data
> > types natively. And your compiler could translate into a fitting P4
> > fixedpoint math library. Or extern libraries, such as for actual floating
> > point calculation. Or even skip mapping into P4 completely and creating
> > better code directly mapping to the underlying platform features.
> >
> > > TM is still an open issue to work on.
> >
> > There is good work happening for high-speed PIFO type FPGA forwarding code.
> > I think some form of extended PIFO functionality could become the basis for
> > programmable TM in the future. Still has to be seen how well it would
> > possible
> > to efficiently PoC this on lower end CPU forwarders. If this direction pans
> > out, then the forwarding plane impact is just the compute of the TM
> > parameters such as rank/timestamps - and processing on dequeueing.
> >
> > Cheers
> >     Toerless
> >
> > >
> > > Hemant
> > >
> > > -----Original Message-----
> > > From: 'Toerless Eckert' <[e-mail]>
> > > Sent: Friday, May 19, 2023 12:33 PM
> > > To: Haoyu Song <[e-mail]>
> > > Cc: [e-mail]; 'Hesham ElBakoury' <[e-mail]>;
> > [e-mail]; 'Bernier, Daniel' <[e-mail]>;
> > 'Marie-Jose Montpetit' <[e-mail]>; 'coin' <[e-mail]>;
> > 'coinrg-chairs' <[e-mail]>
> > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > >
> > > Right.
> > >
> > > For high speed router/switch (>= 100gbps/interface) validation:
> > >   P4 implementations on Tofino are great as proof of concept - if it can
> > be done.
> > >   If P4 implementation on Tofino does not work, other proof points are
> > needed,
> > >
> > > For medium speed "software forwarding" validation:
> > >   P4 creates too many limitations. All CPU have float, and/or SMDI,
> > and/or
> > >   more flexible memory access options and likely several other aspects
> > not
> > >   easily utilized by P4.
> > >
> > > For generic specification:
> > >   IMHO above limitations are at the core of why P4 is insufficient.
> > >   I very much like several structural aspects of P4, they would be great
> > >   starting points for general purpose spec. Such as header parsing specs
> > >   (need to be extended for variable length though).
> > >   Aka: Anything we can use from P4 which is more/better declarative than
> > >   C/C++ is IMHO useful. Just lets not introduce constraints: Validation
> > on
> > >   different type of forwarding platforms is different from a single
> > >   specification.
> > >   And we are also missing good DSL spec options for TM i fear...
> > >
> > > Cheers
> > >     Toerless
> > >
> > > On Fri, May 19, 2023 at 04:02:57PM +0000, Haoyu Song wrote:
> > > > I think the discussion is off the point. We are discussing whether or
> > not P4 is suitable for a standard specification language for general
> > dataplane processing and forwarding applications. To fulfill that purpose,
> > it needs to be target/architecture independent; it needs to be concise and
> > expressive to cover various dataplane functions which may not be
> > conceivable today (e.g., in-network computing applications).
> > > > Unfortunately, I don't think P4 can meet these requirements now
> > > > (imagine all the pseudo code in RFCs are translated to P4). Of course,
> > > > it can be limited to just support certain functions, e.g., to document
> > > > the header formats and parsing process (but here I don't see the
> > > > advantage of P4 either)
> > > >
> > > > Haoyu
> > > >
> > > > -----Original Message-----
> > > > From: [e-mail] <[e-mail]>
> > > > Sent: Thursday, May 18, 2023 6:19 PM
> > > > To: 'Toerless Eckert' <[e-mail]>
> > > > Cc: Haoyu Song <[e-mail]>; 'Hesham ElBakoury'
> > > > <[e-mail]>; [e-mail]; 'Bernier, Daniel'
> > > > <[e-mail]>; 'Marie-Jose Montpetit'
> > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > <[e-mail]>
> > > > Subject: RE: [Coin] Fwd: The Future of P4, Revisited
> > > >
> > > > Toerless,
> > > >
> > > > Only externs defined in core.p4 are in a library. Externs defined in
> > an architecture include file, e.g., v1model.p4 are implemented in
> > target-specific back end and are extensions. An example of a target extern
> > is vpp accelerator in Octeon 10. Neither P4-16 spec nor PNA support
> > floating point and neither does my compiler. Floating point in used be
> > AI/ML. The float point numbers would have to be converted to integer.
> > Google's Tensor chip also converts float to integer for ML.
> > > >
> > > > If you want to work with the linux kernel, see the ebpf architecture
> > include file,
> > https://github.com/p4lang/p4c/blob/main/p4include/ebpf_model.p4. Doesn't
> > the file have externs which have totally abstracted linux kernel data
> > structs and functions?
> > > >
> > > > Hemant
> > > >
> > > > -----Original Message-----
> > > > From: Coin <[e-mail]> On Behalf Of 'Toerless Eckert'
> > > > Sent: Thursday, May 18, 2023 8:23 PM
> > > > To: [e-mail]
> > > > Cc: 'Haoyu Song' <[e-mail]>; 'Hesham ElBakoury'
> > > > <[e-mail]>; [e-mail]; 'Bernier, Daniel'
> > > > <[e-mail]>; 'Marie-Jose Montpetit'
> > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > <[e-mail]>
> > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > >
> > > > Hemant,
> > > >
> > > > Externs and functions are not an extension to the language.
> > > > They are really just libraries.
> > > >
> > > > Syntactic changes/extensions such as for parsing/deparsing, new types,
> > such as float/double - those would be language extensions.
> > > >
> > > > This makes the use of P4 cumbersome on any platform that can do more
> > than the worst-case-chip against which the P4 language is designed (aka:
> > Tofino).
> > > >
> > > > For example on your OCTEON P4 compiler: The ARM A72 does support
> > floating point numbers. If you implemented support for them via the
> > mechanisms supported by P4, the way i understood it, you could still not
> > write simple P4 programs for OCTEON then like:
> > > >
> > > >     float a, b, c
> > > >     a = b * c / 3.2;
> > > >
> > > > You may have a set of externs/functions through which you could do
> > this, but it would look a lot more ugly.
> > > >
> > > > Right ? If not, pls. explain - i would be happy!
> > > >
> > > > And Haoyu's argument was that when we want to suggest to use P4 as the
> > reference specification or base forwarding programming in the linux kernel,
> > then this need to hide stuff outside of P4 written externs and functions
> > does limit P4's useful as a complete specification and reference code
> > language.
> > > >
> > > > Cheers
> > > >     Toerless
> > > >
> > > > On Thu, May 18, 2023 at 04:56:31PM -0400, [e-mail] wrote:
> > > > > Why do you care for internal behavior of checksum when the P4 api is
> > able to provide data to compute checksum.
> > > > >
> > > > > https://github.com/p4lang/p4c/blob/main/p4include/v1model.p4#L505
> > > > >
> > > > > Hemant
> > > > >
> > > > > -----Original Message-----
> > > > > From: Coin <[e-mail]> On Behalf Of Haoyu Song
> > > > > Sent: Thursday, May 18, 2023 3:30 PM
> > > > > To: 'Toerless Eckert' <[e-mail]>; [e-mail]
> > > > > Cc: 'Hesham ElBakoury' <[e-mail]>; [e-mail];
> > > > > 'Bernier, Daniel' <[e-mail]>; 'Marie-Jose Montpetit'
> > > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > > <[e-mail]>
> > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > >
> > > > > Toerless,
> > > > >
> > > > > This is exactly the problem: you can't use the core language to
> > describe an arbitrary dataplane function.
> > > > > "Extern" can be a blackbox but what need is the spec for what's in
> > the blackbox.
> > > > >
> > > > > For P4 spec: " Extern objects are architecture-specific constructs
> > that can be manipulated by P4 programs through well-defined APIs, but whose
> > internal behavior is hard-wired (e.g., checksum units) and hence not
> > programmable using P4"
> > > > >
> > > > > Haoyu
> > > > >
> > > > > -----Original Message-----
> > > > > From: 'Toerless Eckert' <[e-mail]>
> > > > > Sent: Thursday, May 18, 2023 12:15 PM
> > > > > To: [e-mail]
> > > > > Cc: Haoyu Song <[e-mail]>; 'Hesham ElBakoury'
> > > > > <[e-mail]>; [e-mail]; 'Bernier, Daniel'
> > > > > <[e-mail]>; 'Marie-Jose Montpetit'
> > > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > > <[e-mail]>
> > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > >
> > > > > On Thu, May 18, 2023 at 02:19:12PM -0400, [e-mail] wrote:
> > > > > > A specific P4 architecture, e.g., PNA, may override the base P4
> > spec. PNA supports writing table entry by data plane. Using registers in P4
> > is a bit clunky but state can be maintained. Anyone is welcome to present a
> > better solution to the P4 forums.
> > > > >
> > > > > So far, my limited experience has been, that the language itself has
> > rather evolved over its series of spec to further reduce the base language
> > functionality and move any improvemenets into implementation extensions
> > ("extern") calls, so that no possible hardware would not be able to not
> > support any base language feature.
> > > > >
> > > > > This is a good strategy to gain more acceptance in the industry, but
> > > > > not to get to the easiest to use forwarding plane behavioral
> > > > > definition language
> > > > >
> > > > > For exmple, i would need axtual syntactical language extensions to
> > define a good way to parse andd deparsing variable length header fields.
> > This can not just be done with extern calls.
> > > > >
> > > > > Has P4 adopted the approach to have optional language feature
> > extensions ?
> > > > > Sorry. Did not have time to follow the spec evolution in detail.
> > > > >
> > > > > Cheers
> > > > >     Toerless
> > > > > > Hemant
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Haoyu Song <[e-mail]>
> > > > > > Sent: Thursday, May 18, 2023 1:56 PM
> > > > > > To: Hesham ElBakoury <[e-mail]>
> > > > > > Cc: [e-mail]; Toerless Eckert <[e-mail]>;
> > > > > > hemant=[e-mail] <[e-mail]>; Bernier,
> > > > > > Daniel <[e-mail]>; Marie-Jose Montpetit
> > > > > > <[e-mail]>; coin <[e-mail]>; coinrg-chairs
> > > > > > <[e-mail]>
> > > > > > Subject: RE: [Coin] Fwd: The Future of P4, Revisited
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Hesham,
> > > > > >
> > > > > >
> > > > > >
> > > > > > Please see P4 spec section 6.5.2 for the current “stateful”
> > support.
> > > > > > So far the “table” element, which is the most important construct
> > > > > > for a P4 program,  is not stateful (i.e., dataplane writable)
> > > > > >
> > > > > > I think it’s a direction to extend P4 for better stateful support
> > (part of my recent research). At least now it’s unnatural and difficult to
> > describe many stateful functions.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Haoyu
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Coin <[e-mail] <mailto:[e-mail]>
> > >
> > > > > > On Behalf Of Hesham ElBakoury
> > > > > > Sent: Thursday, May 18, 2023 10:42 AM
> > > > > > To: Haoyu Song <[e-mail]
> > > > > > <mailto:[e-mail]> >
> > > > > > Cc: [e-mail] <mailto:[e-mail]> ; Toerless
> > > > > > Eckert <[e-mail] <mailto:[e-mail]> >;
> > > > > > hemant=[e-mail]
> > > > > > <mailto:hemant=[e-mail]> ; Bernier, Daniel
> > > > > > <[e-mail] <mailto:[e-mail]> >;
> > > > > > Marie-Jose Montpetit <[e-mail]
> > > > > > <mailto:[e-mail]> >; coin <[e-mail]
> > > > > > <mailto:[e-mail]> >; coinrg-chairs <[e-mail]
> > > > > > <mailto:[e-mail]> >
> > > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Haoyu,
> > > > > >
> > > > > > Actually, P4 supports dataplane-modifiable tables -- see PNA.
> > > > > >
> > > > > >
> > > > > >
> > > > > > More generally, the *language* is fully extensible. You can have
> > whatever architecture and state externs you want. So one needs to be
> > careful to separate language (non) limitations from target limitations.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hesham
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, May 18, 2023, 10:15 AM Haoyu Song <
> > [e-mail] <mailto:[e-mail]> > wrote:
> > > > > >
> > > > > > Hi Hesham,
> > > > > >
> > > > > >
> > > > > >
> > > > > > I said “it’s very limited”. P4 table is only readable (i.e., not
> > writable) by dataplane, so it basically eliminates any dataplane stateful
> > function that need to use tables. The stateful function can only use
> > register arrays but unfortunately registers are local to a pipeline stage
> > so the state update logic must be very simple and can be finish in a single
> > stage. What you said “event” must be something very simple. Use case such
> > as stateful load balancer can’t be implemented by P4.
> > > > > >
> > > > > >
> > > > > >
> > > > > > There are some works addressing the issue. FlowBlaze is the most
> > recent one, which needs a new chip architecture but it is still limited to
> > simple stateful functions which can be realized in a single pipeline stage.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Haoyu
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Hesham ElBakoury <[e-mail]
> > > > > > <mailto:[e-mail]> >
> > > > > > Sent: Thursday, May 18, 2023 10:00 AM
> > > > > > To: Haoyu Song <[e-mail]
> > > > > > <mailto:[e-mail]> >
> > > > > > Cc: [e-mail] <mailto:[e-mail]> ; Toerless
> > > > > > Eckert <[e-mail] <mailto:[e-mail]> >;
> > > > > > hemant=[e-mail]
> > > > > > <mailto:[e-mail]>
> > > > > > ; Bernier, Daniel <[e-mail]
> > > > > > <mailto:[e-mail]> >; Marie-Jose Montpetit
> > > > > > <[e-mail] <mailto:[e-mail]> >; coin
> > > > > > <[e-mail] <mailto:[e-mail]> >; coinrg-chairs
> > > > > > <[e-mail] <mailto:[e-mail]> >
> > > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Haoyu,
> > > > > >
> > > > > > You say: "P4 has very limited support for stateful processing"
> > > > > >
> > > > > >
> > > > > >
> > > > > > I am not sure I can agree with your characterization of P4.
> > Perhaps can you elaborate on this.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Hesham
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, May 18, 2023, 9:19 AM Haoyu Song <[e-mail]
> > <mailto:[e-mail]> > wrote:
> > > > > >
> > > > > > Interesting discussion. See my comments below  [HS]
> > > > > >
> > > > > > Haoyu
> > > > > >
> > > > > > > For example, in the multicast drafts i write, we use
> > > > > > > C-pseudocode to specify behavior, but we do attempt to implemnt
> > on Tofino in P4.
> > > > > > > Should we really use P4 code for the RFC specs... ? (much longer
> > > > > > > than C Pseudocode). Aka: quite selfish (but IETF relevant ;-)
> > reason to highlight this point.
> > > > > >
> > > > > > [EH]: This is an area I'm very interested in. Having a
> > standardized and formal language to describe protocols and behavior can
> > bring a lot of functionality and benefits to the IETF.
> > > > > > My initial thinking is that having such a blueprint, the IETF
> > could generate tools to create a reference implementation that can be used
> > for interoperability purposes therefore decreasing time to test and
> > implement protocols and therefore RFC publications.
> > > > > >
> > > > > > [HS] P4 can only describe dataplane behaviors, so any control
> > plane stuff is out of scope. For dataplane, if it's used to describe header
> > format, it's not better than the "struct" in C. The language uses the
> > match-action table abstraction with an implication of pipeline
> > implementation which may make it cumbersome or even impossible to describe
> > the  behavior (e.g.,  P4 has very limited support for stateful processing).
> > In general, I don't think P4 at its current form can undertake the role for
> > formal protocol specification.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > ---
> > > > > [e-mail]
> > > > > --
> > > > > Coin mailing list
> > > > > [e-mail]
> > > > > https://www.irtf.org/mailman/listinfo/coin
> > > >
> > > >
> > > >
> > > > --
> > > > ---
> > > > [e-mail]
> > > >
> > > > --
> > > > Coin mailing list
> > > > [e-mail]
> > > > https://www.irtf.org/mailman/listinfo/coin
> > > > --
> > > > Coin mailing list
> > > > [e-mail]
> > > > https://www.irtf.org/mailman/listinfo/coin
> > >
> > > --
> > > ---
> > > [e-mail]
> >
> >
> >
> > --
> > ---
> > [e-mail]
> >

-- 
---
[e-mail]