[Coin] The Price for Programmability in the Software Data Plane: The Vendor Perspective

[Hesham ElBakoury <[e-mail]>]

This JSAC paper was published in 2018 to describe Ericsson experimentation
with software  programmable data plane [
http://lendulet.tmit.bme.hu/~retvari/publications/jsac_2018.pdf]

I think you can implement data plane in X86 server with less cost and
energy consumption compared to Tofino for Tbps or less performance. But of
course software data plane has the shortcomings that are described in this
paper. Also P4 lacks features such as Traffic management which if it can't
be implemented adequately using "extern"[1[, we need to modify the P4
langauge.

Is anyone aware of recent studies that compare using empirical analysis
between different programmable data plane solutions in terms of
performance, cost, energy consumption and scalability?

Thanks
Hesham
[1] A High-Speed, Scalable, and Programmable Traffic Manager Architecture
for Flow-Based Networking, I
https://core.ac.uk/download/pdf/401883686.pdf





On Fri, May 19, 2023, 4:18 PM <[e-mail]> wrote:

> DPDK uses HQF and since VPP uses DPDK for packet i/o, VPP also supports
> HQF. Programming HQF running on an asic is very complex to program.
>
> I am personally not going to use the Tofino - any asic feature development
> feature development works too slow for me.
> Vpp code I referred to is not ugly for me - cache alignment is in vpp
> infra which developers of new plugins just use via api. Reading P4 or C
> code is no different to me. With Octeon, I am not at the mercy of an asic
> vendor because Octeon lets me use DPDK or VPP to program whatever I want.
>
> Cisco has a large swath of P4-16 code running on their CiscoONE asic.
> Mellanox has P4 support. Intel FPGA is P4-programmable.
>
> Hemant
>
> -----Original Message-----
> From: 'Toerless Eckert' <[e-mail]>
> Sent: Friday, May 19, 2023 3:22 PMogra
> mTo: [e-mail]
> Cc: 'Haoyu Song' <[e-mail]>; 'Hesham ElBakoury' <
> [e-mail]>; [e-mail]; 'Bernier, Daniel' <
> [e-mail]>; 'Marie-Jose Montpetit' <[e-mail]>;
> 'coin' <[e-mail]>; 'coinrg-chairs' <[e-mail]>
> Subject: Re: [Coin] Fwd: The Future of P4, Revisited
>
> On Fri, May 19, 2023 at 02:44:15PM -0400, [e-mail] wrote:
> > I wrote parsing TCP options in P4 using varbits in P4 parser for
> > bmv2. Se
> >
> > https://github.com/p4lang/p4c/blob/main/testdata/p4_16_samples/checksu
> > m-l4-bmv2.p4
> Sure, a lot has been done and is possible with VPP. And if you want to
> propse something data plane to IETF and want to claim that it will work for
> cost effective data center switches then you should do a Tofio PoC.
> At least that's one of he reason why i want some of my IETF work to have
> Tofino PoC. But thats independent of what the best specification language
> is. Aka: For something like these TCP header options, CPU C code would be a
> lot easie to read, right?
>
> > in line below.
>
> > Hemant: VPP has a simple rule: fit code in instructions cache and tables
> in data cache and vpp will rock and roll. Use of buffer pools and
> cache-aligned data is common code for x86, ARM, or risc-v.
>
> Sure. That also results in a lot of code uglyness starting from breaking
> processing into the right size chunks to have a good amount of packets to
> form vectors from, and the figure out which CPU can best work with which
> vector length, etc. pp.
>
> > Hemant: I know about PIFO. It cannot track bandwidth across all flows
> like HQF can. This is why I did discard PIFO.
>
> I was suggestingo PIFO as a core building block. I am not quite sure about
> the minimum necessary framework around it. I am not quite sure about the
> details of mapping HQF to PIFO, but it is claimed to be possible, e.g.:
> https://www.kom.tu-darmstadt.de/papers/GRK+21-2.pdf
> "the PIFO queuing concept supports hierarchical queueing, which are
> drained from the root"
>
> There are some queuing concepts not suported by PIFO, so definitely worth
> to better understand those and their relevance.
>
> But i do agree, that HQF is the bread-and-butter queuing requirement for
> all the edge interface deployment casees today, because it is pretty much
> what the industry has standardized on so far - and every advanced TM chip
> has pretty much HQF and unfortunately often also only HQF.
>
> How do we make more researchers interested in TM ? It is architecturally
> still so much unexplored than any of the other compute aspects in
> forwarding planes, even though it did have becauseof PIFO since 2016 some
> good influx of work...
>
> Cheers
>     Toerless
>
> > Hemant
> >
> >
> > Cheers
> >     Toerless
> >
> > >
> > > Hemant
> > >
> > > -----Original Message-----
> > > From: 'Toerless Eckert' <[e-mail]>
> > > Sent: Friday, May 19, 2023 12:33 PM
> > > To: Haoyu Song <[e-mail]>
> > > Cc: [e-mail]; 'Hesham ElBakoury' <[e-mail]>;
> > > [e-mail]; 'Bernier, Daniel' <[e-mail]>;
> > > 'Marie-Jose Montpetit' <[e-mail]>; 'coin'
> > > <[e-mail]>; 'coinrg-chairs' <[e-mail]>
> > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > >
> > > Right.
> > >
> > > For high speed router/switch (>= 100gbps/interface) validation:
> > >   P4 implementations on Tofino are great as proof of concept - if it
> can be done.
> > >   If P4 implementation on Tofino does not work, other proof points
> > > are needed,
> > >
> > > For medium speed "software forwarding" validation:
> > >   P4 creates too many limitations. All CPU have float, and/or SMDI,
> and/or
> > >   more flexible memory access options and likely several other aspects
> not
> > >   easily utilized by P4.
> > >
> > > For generic specification:
> > >   IMHO above limitations are at the core of why P4 is insufficient.
> > >   I very much like several structural aspects of P4, they would be
> great
> > >   starting points for general purpose spec. Such as header parsing
> specs
> > >   (need to be extended for variable length though).
> > >   Aka: Anything we can use from P4 which is more/better declarative
> than
> > >   C/C++ is IMHO useful. Just lets not introduce constraints:
> Validation on
> > >   different type of forwarding platforms is different from a single
> > >   specification.
> > >   And we are also missing good DSL spec options for TM i fear...
> > >
> > > Cheers
> > >     Toerless
> > >
> > > On Fri, May 19, 2023 at 04:02:57PM +0000, Haoyu Song wrote:
> > > > I think the discussion is off the point. We are discussing whether
> or not P4 is suitable for a standard specification language for general
> dataplane processing and forwarding applications. To fulfill that purpose,
> it needs to be target/architecture independent; it needs to be concise and
> expressive to cover various dataplane functions which may not be
> conceivable today (e.g., in-network computing applications).
> > > > Unfortunately, I don't think P4 can meet these requirements now
> > > > (imagine all the pseudo code in RFCs are translated to P4). Of
> > > > course, it can be limited to just support certain functions, e.g.,
> > > > to document the header formats and parsing process (but here I
> > > > don't see the advantage of P4 either)
> > > >
> > > > Haoyu
> > > >
> > > > -----Original Message-----
> > > > From: [e-mail] <[e-mail]>
> > > > Sent: Thursday, May 18, 2023 6:19 PM
> > > > To: 'Toerless Eckert' <[e-mail]>
> > > > Cc: Haoyu Song <[e-mail]>; 'Hesham ElBakoury'
> > > > <[e-mail]>; [e-mail]; 'Bernier, Daniel'
> > > > <[e-mail]>; 'Marie-Jose Montpetit'
> > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > <[e-mail]>
> > > > Subject: RE: [Coin] Fwd: The Future of P4, Revisited
> > > >
> > > > Toerless,
> > > >
> > > > Only externs defined in core.p4 are in a library. Externs defined in
> an architecture include file, e.g., v1model.p4 are implemented in
> target-specific back end and are extensions. An example of a target extern
> is vpp accelerator in Octeon 10. Neither P4-16 spec nor PNA support
> floating point and neither does my compiler. Floating point in used be
> AI/ML. The float point numbers would have to be converted to integer.
> Google's Tensor chip also converts float to integer for ML.
> > > >
> > > > If you want to work with the linux kernel, see the ebpf architecture
> include file,
> https://github.com/p4lang/p4c/blob/main/p4include/ebpf_model.p4. Doesn't
> the file have externs which have totally abstracted linux kernel data
> structs and functions?
> > > >
> > > > Hemant
> > > >
> > > > -----Original Message-----
> > > > From: Coin <[e-mail]> On Behalf Of 'Toerless Eckert'
> > > > Sent: Thursday, May 18, 2023 8:23 PM
> > > > To: [e-mail]
> > > > Cc: 'Haoyu Song' <[e-mail]>; 'Hesham ElBakoury'
> > > > <[e-mail]>; [e-mail]; 'Bernier, Daniel'
> > > > <[e-mail]>; 'Marie-Jose Montpetit'
> > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > <[e-mail]>
> > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > >
> > > > Hemant,
> > > >
> > > > Externs and functions are not an extension to the language.
> > > > They are really just libraries.
> > > >
> > > > Syntactic changes/extensions such as for parsing/deparsing, new
> types, such as float/double - those would be language extensions.
> > > >
> > > > This makes the use of P4 cumbersome on any platform that can do more
> than the worst-case-chip against which the P4 language is designed (aka:
> Tofino).
> > > >
> > > > For example on your OCTEON P4 compiler: The ARM A72 does support
> floating point numbers. If you implemented support for them via the
> mechanisms supported by P4, the way i understood it, you could still not
> write simple P4 programs for OCTEON then like:
> > > >
> > > >     float a, b, c
> > > >     a = b * c / 3.2;
> > > >
> > > > You may have a set of externs/functions through which you could do
> this, but it would look a lot more ugly.
> > > >
> > > > Right ? If not, pls. explain - i would be happy!
> > > >
> > > > And Haoyu's argument was that when we want to suggest to use P4 as
> the reference specification or base forwarding programming in the linux
> kernel, then this need to hide stuff outside of P4 written externs and
> functions does limit P4's useful as a complete specification and reference
> code language.
> > > >
> > > > Cheers
> > > >     Toerless
> > > >
> > > > On Thu, May 18, 2023 at 04:56:31PM -0400, [e-mail] wrote:
> > > > > Why do you care for internal behavior of checksum when the P4 api
> is able to provide data to compute checksum.
> > > > >
> > > > > https://github.com/p4lang/p4c/blob/main/p4include/v1model.p4#L50
> > > > > 5
> > > > >
> > > > > Hemant
> > > > >
> > > > > -----Original Message-----
> > > > > From: Coin <[e-mail]> On Behalf Of Haoyu Song
> > > > > Sent: Thursday, May 18, 2023 3:30 PM
> > > > > To: 'Toerless Eckert' <[e-mail]>; [e-mail]
> > > > > Cc: 'Hesham ElBakoury' <[e-mail]>;
> > > > > [e-mail]; 'Bernier, Daniel' <[e-mail]>;
> 'Marie-Jose Montpetit'
> > > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > > <[e-mail]>
> > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > >
> > > > > Toerless,
> > > > >
> > > > > This is exactly the problem: you can't use the core language to
> describe an arbitrary dataplane function.
> > > > > "Extern" can be a blackbox but what need is the spec for what's in
> the blackbox.
> > > > >
> > > > > For P4 spec: " Extern objects are architecture-specific constructs
> that can be manipulated by P4 programs through well-defined APIs, but whose
> internal behavior is hard-wired (e.g., checksum units) and hence not
> programmable using P4"
> > > > >
> > > > > Haoyu
> > > > >
> > > > > -----Original Message-----
> > > > > From: 'Toerless Eckert' <[e-mail]>
> > > > > Sent: Thursday, May 18, 2023 12:15 PM
> > > > > To: [e-mail]
> > > > > Cc: Haoyu Song <[e-mail]>; 'Hesham ElBakoury'
> > > > > <[e-mail]>; [e-mail]; 'Bernier, Daniel'
> > > > > <[e-mail]>; 'Marie-Jose Montpetit'
> > > > > <[e-mail]>; 'coin' <[e-mail]>; 'coinrg-chairs'
> > > > > <[e-mail]>
> > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > >
> > > > > On Thu, May 18, 2023 at 02:19:12PM -0400, [e-mail] wrote:
> > > > > > A specific P4 architecture, e.g., PNA, may override the base P4
> spec. PNA supports writing table entry by data plane. Using registers in P4
> is a bit clunky but state can be maintained. Anyone is welcome to present a
> better solution to the P4 forums.
> > > > >
> > > > > So far, my limited experience has been, that the language itself
> has rather evolved over its series of spec to further reduce the base
> language functionality and move any improvemenets into implementation
> extensions ("extern") calls, so that no possible hardware would not be able
> to not support any base language feature.
> > > > >
> > > > > This is a good strategy to gain more acceptance in the industry,
> > > > > but not to get to the easiest to use forwarding plane behavioral
> > > > > definition language
> > > > >
> > > > > For exmple, i would need axtual syntactical language extensions to
> define a good way to parse andd deparsing variable length header fields.
> This can not just be done with extern calls.
> > > > >
> > > > > Has P4 adopted the approach to have optional language feature
> extensions ?
> > > > > Sorry. Did not have time to follow the spec evolution in detail.
> > > > >
> > > > > Cheers
> > > > >     Toerless
> > > > > > Hemant
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Haoyu Song <[e-mail]>
> > > > > > Sent: Thursday, May 18, 2023 1:56 PM
> > > > > > To: Hesham ElBakoury <[e-mail]>
> > > > > > Cc: [e-mail]; Toerless Eckert <[e-mail]>;
> > > > > > hemant=[e-mail] <[e-mail]>; Bernier,
> > > > > > Daniel <[e-mail]>; Marie-Jose Montpetit
> > > > > > <[e-mail]>; coin <[e-mail]>; coinrg-chairs
> > > > > > <[e-mail]>
> > > > > > Subject: RE: [Coin] Fwd: The Future of P4, Revisited
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Hesham,
> > > > > >
> > > > > >
> > > > > >
> > > > > > Please see P4 spec section 6.5.2 for the current “stateful”
> support.
> > > > > > So far the “table” element, which is the most important
> > > > > > construct for a P4 program,  is not stateful (i.e., dataplane
> > > > > > writable)
> > > > > >
> > > > > > I think it’s a direction to extend P4 for better stateful
> support (part of my recent research). At least now it’s unnatural and
> difficult to describe many stateful functions.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Haoyu
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Coin <[e-mail]
> > > > > > <mailto:[e-mail]>
> > > > > > > On Behalf Of Hesham ElBakoury
> > > > > > Sent: Thursday, May 18, 2023 10:42 AM
> > > > > > To: Haoyu Song <[e-mail]
> > > > > > <mailto:[e-mail]> >
> > > > > > Cc: [e-mail] <mailto:[e-mail]> ;
> > > > > > Toerless Eckert <[e-mail] <mailto:[e-mail]> >;
> > > > > > hemant=[e-mail]
> > > > > > <mailto:hemant=[e-mail]> ; Bernier, Daniel
> > > > > > <[e-mail] <mailto:[e-mail]> >;
> > > > > > Marie-Jose Montpetit <[e-mail]
> > > > > > <mailto:[e-mail]> >; coin <[e-mail]
> > > > > > <mailto:[e-mail]> >; coinrg-chairs
> > > > > > <[e-mail] <mailto:[e-mail]> >
> > > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Haoyu,
> > > > > >
> > > > > > Actually, P4 supports dataplane-modifiable tables -- see PNA.
> > > > > >
> > > > > >
> > > > > >
> > > > > > More generally, the *language* is fully extensible. You can have
> whatever architecture and state externs you want. So one needs to be
> careful to separate language (non) limitations from target limitations.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hesham
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, May 18, 2023, 10:15 AM Haoyu Song <
> [e-mail] <mailto:[e-mail]> > wrote:
> > > > > >
> > > > > > Hi Hesham,
> > > > > >
> > > > > >
> > > > > >
> > > > > > I said “it’s very limited”. P4 table is only readable (i.e., not
> writable) by dataplane, so it basically eliminates any dataplane stateful
> function that need to use tables. The stateful function can only use
> register arrays but unfortunately registers are local to a pipeline stage
> so the state update logic must be very simple and can be finish in a single
> stage. What you said “event” must be something very simple. Use case such
> as stateful load balancer can’t be implemented by P4.
> > > > > >
> > > > > >
> > > > > >
> > > > > > There are some works addressing the issue. FlowBlaze is the most
> recent one, which needs a new chip architecture but it is still limited to
> simple stateful functions which can be realized in a single pipeline stage.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Haoyu
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Hesham ElBakoury <[e-mail]
> > > > > > <mailto:[e-mail]> >
> > > > > > Sent: Thursday, May 18, 2023 10:00 AM
> > > > > > To: Haoyu Song <[e-mail]
> > > > > > <mailto:[e-mail]> >
> > > > > > Cc: [e-mail] <mailto:[e-mail]> ;
> > > > > > Toerless Eckert <[e-mail] <mailto:[e-mail]> >;
> > > > > > hemant=[e-mail]
> > > > > > <mailto:[e-mail]>
> > > > > > ; Bernier, Daniel <[e-mail]
> > > > > > <mailto:[e-mail]> >; Marie-Jose Montpetit
> > > > > > <[e-mail] <mailto:[e-mail]> >; coin
> > > > > > <[e-mail] <mailto:[e-mail]> >; coinrg-chairs
> > > > > > <[e-mail] <mailto:[e-mail]> >
> > > > > > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Haoyu,
> > > > > >
> > > > > > You say: "P4 has very limited support for stateful processing"
> > > > > >
> > > > > >
> > > > > >
> > > > > > I am not sure I can agree with your characterization of P4.
> Perhaps can you elaborate on this.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Hesham
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, May 18, 2023, 9:19 AM Haoyu Song <
> [e-mail] <mailto:[e-mail]> > wrote:
> > > > > >
> > > > > > Interesting discussion. See my comments below  [HS]
> > > > > >
> > > > > > Haoyu
> > > > > >
> > > > > > > For example, in the multicast drafts i write, we use
> > > > > > > C-pseudocode to specify behavior, but we do attempt to
> implemnt on Tofino in P4.
> > > > > > > Should we really use P4 code for the RFC specs... ? (much
> > > > > > > longer than C Pseudocode). Aka: quite selfish (but IETF
> relevant ;-) reason to highlight this point.
> > > > > >
> > > > > > [EH]: This is an area I'm very interested in. Having a
> standardized and formal language to describe protocols and behavior can
> bring a lot of functionality and benefits to the IETF.
> > > > > > My initial thinking is that having such a blueprint, the IETF
> could generate tools to create a reference implementation that can be used
> for interoperability purposes therefore decreasing time to test and
> implement protocols and therefore RFC publications.
> > > > > >
> > > > > > [HS] P4 can only describe dataplane behaviors, so any control
> plane stuff is out of scope. For dataplane, if it's used to describe header
> format, it's not better than the "struct" in C. The language uses the
> match-action table abstraction with an implication of pipeline
> implementation which may make it cumbersome or even impossible to describe
> the  behavior (e.g.,  P4 has very limited support for stateful processing).
> In general, I don't think P4 at its current form can undertake the role for
> formal protocol specification.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > ---
> > > > > [e-mail]
> > > > > --
> > > > > Coin mailing list
> > > > > [e-mail]
> > > > > https://www.irtf.org/mailman/listinfo/coin
> > > >
> > > >
> > > >
> > > > --
> > > > ---
> > > > [e-mail]
> > > >
> > > > --
> > > > Coin mailing list
> > > > [e-mail]
> > > > https://www.irtf.org/mailman/listinfo/coin
> > > > --
> > > > Coin mailing list
> > > > [e-mail]
> > > > https://www.irtf.org/mailman/listinfo/coin
> > >
> > > --
> > > ---
> > > [e-mail]
> >
> >
> >
> > --
> > ---
> > [e-mail]
>
>
>
> --
> ---
> [e-mail]
>