Re: [Coin] Fwd: The Future of P4, Revisited

[[e-mail]]

Why do you need API if P4 is involved? This is one API for P4:

https://github.com/p4lang/p4c/blob/main/p4include/v1model.p4

Then, use library provided by Marvell to program Octeon hw accelerators and you are good to go.

Hemant
 

-----Original Message-----
From: 'Toerless Eckert' <[e-mail]> 
Sent: Monday, May 15, 2023 3:45 PM
To: Hesham ElBakoury <[e-mail]>
Cc: [e-mail]; 'Marie-Jose Montpetit' <[e-mail]>; 'coin' <[e-mail]>; [e-mail]
Subject: Re: [Coin] Fwd: The Future of P4, Revisited

No idea yet as to whether or not Marvel actually makes their API publically available, but Broadcom certainly does not. I looked long and found no researcher that had managed to get access to it. NPL is purely a proprietary solution offered to custsomers of broadcom, and seemingly the degree of information that customers get depends proportionally to the amount of business for broadcom.

Toerless

On Mon, May 15, 2023 at 12:28:04PM -0700, Hesham ElBakoury wrote:
> How you compare Marvel Octeon 10 with Broadcom Trident and Jericho 2 
> devices which use NPL (https://nplang.org).
> 
> Hesham
> 
> > Toerless,
> > 
> > This is the latest network processor which supports use in a switch or nic.
> > 
> > https://www.marvell.com/content/dam/marvell/en/company/media-kit/oct
> > eon-10/marvell-octeon-10-media-deck.pdf
> > 
> > This is your Tofino replacement with hardware available from Marvell and P4 compiler and tools chain available from my company.
> > 
> > Hemant
> > 
> > 
> > -----Original Message-----
> > From: Coin<[e-mail]>  On Behalf Of Toerless Eckert
> > Sent: Monday, May 15, 2023 1:04 PM
> > To: Marie-Jose Montpetit<[e-mail]>
> > Cc: coin<[e-mail]>;[e-mail]
> > Subject: Re: [Coin] Fwd: The Future of P4, Revisited
> > 
> > Thanks, Marie-Jose
> > 
> > IMHO:
> > 
> > - P4 in research short-term:
> >    Researchers can not do real-world relevant PoC with just BMV2, because it has no constraints.
> >    All those papers claiming some problem can be solved with P4 and then just use BMV2 are just
> >    so painful to read/listen-to. We really need a BMV2 with the exact constraints of Tofino,
> >    and given how Intel is deinvesting from Tofino, maybe they would finally be willing to
> >    publish those constraints outside of NDA so this can be implemented into BMV2.
> > 
> > - SmartNICs:
> >    P4 is not the right DSL for SmartNICs even if Intel may say so because they want to
> >    sell SmartNICs and have a brand with P4. SmartNICs are much more flexible, and programming then
> >    in P4 constraints you. If you don't know anything better, try eBPF. That is AFAIK the
> >    most widely adopted DSL today to span general purpose CPU (host/router CPU), and SmartNICs.
> > 
> >    SmartNICs are a great target FPE for forwarding plane features you do not need on every hop,
> >    but for example only on few "can be more expensive" hops. WAN interface of site-edge-routers
> >    (Campus, Home, WAN) for example. But do not try to assume they would be used beyond that
> >    (on every hop == especially on higher aggregation speed links!).
> > 
> > - P4/Coin for higher-than-forwarding-plane features:
> >    With Tofino disappearing, i think we should caution researchers to NOT invest cycles trying to
> >    implement solutions on P4 with the explicit purpose of "abusing" Tofino, aka: Program a
> >    non-forwarding-plane problem solution in P4 in the expectation that a fast P4 exeuction
> >    product like Tofino would be a great alternative/competition to other execution platforms
> >    (SmartNIC, CPU,...). We had several of those great ideas presented/shown in Coin, they
> >    where all about higher than forwarding-plane "compute" problems. But very quickly, with
> >    Tofino aging, i am sure CPU or SmartNIC slutions for the same problems can easily be
> >    shown to be more cost-effective.
> > 
> > - Long-term forwarding plane research:
> >    The rejection of the industry (outside Intel) to support P4 for researchers on their router/switch
> >    forwarding planes (for a decade now) should primarily be a trigger for research funding
> >    such as from NSF/EU to better enable long-term forwarding plane research. IMHO there
> >    should simply be a single well funded effort for a Network Programming Element (NPE) equivalent
> >    to RISC-V. Aka: fully open source architecture and instruction set for a research NPE,
> >    competitive to industry products. There are lot of other benefits from such an effort
> >    that could help make that happen.
> > 
> > Cheers
> >      Toerless
> > 
> > On Mon, May 15, 2023 at 03:33:42AM -0700, Marie-Jose Montpetit wrote:
> > > Interesting article for out community.
> > > 
> > > mjm
> > > 
> > > Marie-José Montpetit, Ph.D.
> > > [e-mail]
> > > 
> > > 
> > > 
> > > From: Larry Peterson from Systems Approach 
> > > <[e-mail]>  <[e-mail]>
> > > Reply: Larry Peterson from Systems Approach
> > > <reply+1zsws7&rqag0&&3ca860fd966ac1a0bf575ed48379b6645b7df5edc0903
> > > e4c5
> > > [e-mail]>
> > > <reply+1zsws7&rqag0&&3ca860fd966ac1a0bf575ed48379b6645b7df5edc0903
> > > e4c5
> > > [e-mail]>
> > > Date: May 15, 2023 at 3:55:52 AM
> > > To:[e-mail]  <[e-mail]> 
> > > <[e-mail]>
> > > Subject:  The Future of P4, Revisited
> > > 
> > > The P4 Workshop was a couple weeks ago, and as General Chair, I 
> > > went into it with a fair amount of trepidation. My concern was 
> > > that Intel’s announcement earlier this year that they’re 
> > > cancelling development of the Tofino 3 switching chip would have a 
> > > chilling effect, not only on the Workshop, but also on the future 
> > > of P4. That concern has been voiced in several forums  ‌ ‌ ‌ ‌ ‌ ‌ 
> > > ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 
> > > ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 
> > > ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 
> > > ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 
> > > ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 
> > > ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ Open in 
> > > app
> > > <https://open.substack.com/pub/systemsapproach/p/the-future-of-p4-
> > > one- perspective?utm_source=email&redirect=app-store>
> > > or online
> > > <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwc
> > > m9hY
> > > 2guc3Vic3RhY2suY29tL3AvdGhlLWZ1dHVyZS1vZi1wNC1vbmUtcGVyc3BlY3RpdmU
> > > _dG9 
> > > rZW49ZXlKMWMyVnlYMmxrSWpvME5qVTNOakl5TkN3aWNHOXpkRjlwWkNJNk1USXdOa
> > > kF4T 
> > > lRreExDSnBZWFFpT2pFMk9EUXhNemN6TkRrc0ltVjRjQ0k2TVRZNE5qY3lPVE0wT1N
> > > 3aWF 
> > > YTnpJam9pY0hWaUxUSTRNemswTXlJc0luTjFZaUk2SW5CdmMzUXRjbVZoWTNScGIyN
> > > GlmU 
> > > S50UVhzNmRNSG1GRWdMS0I0dlVHNHVHSlRVU0UwTjdVZEFIMXFzajg2bUNBIiwicCI
> > > 6MTI 
> > > wNjAxNTkxLCJzIjoyODM5NDMsImYiOnRydWUsInUiOjQ2NTc2MjI0LCJpYXQiOjE2O
> > > DQxM 
> > > zczNDksImV4cCI6MTY4NjcyOTM0OSwiaXNzIjoicHViLTAiLCJzdWIiOiJsaW5rLXJ
> > > lZGl yZWN0In0.6L7-AmSfpdLfVXnKIHzqDqYX9u_tCmRVPJC2N9oP8gU?>
> > > The Future of P4, Revisited
> > > <https://substack.com/app-link/post?publication_id=283943&post_id=
> > > 1206 
> > > 01591&utm_source=post-email-title&isFreemail=true&token=eyJ1c2VyX2
> > > lkIj
> > > o0NjU3NjIyNCwicG9zdF9pZCI6MTIwNjAxNTkxLCJpYXQiOjE2ODQxMzczNDksImV4
> > > cCI6 
> > > MTY4NjcyOTM0OSwiaXNzIjoicHViLTI4Mzk0MyIsInN1YiI6InBvc3QtcmVhY3Rpb2
> > > 4ifQ .tQXs6dMHmFEgLKB4vUG4uGJTUSE0N7UdAH1qsj86mCA>
> > > 
> > > Larry Peterson
> > > <https://substack.com/redirect/ff29b5a3-7ea9-4b83-b888-b0c517b2bed
> > > d?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > May 15
> > > <https://substack.com/redirect/ff29b5a3-7ea9-4b83-b888-b0c517b2bed
> > > d?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > 
> > > <https://substack.com/app-link/post?publication_id=283943&post_id=
> > > 1206
> > > 01591&utm_source=substack&isFreemail=true&submitLike=true&token=ey
> > > J1c2 
> > > VyX2lkIjo0NjU3NjIyNCwicG9zdF9pZCI6MTIwNjAxNTkxLCJyZWFjdGlvbiI6IuKd
> > > pCIs 
> > > ImlhdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMjgzOT
> > > QzIi 
> > > wic3ViIjoicmVhY3Rpb24ifQ.vN6AE7iLQvKkv98V0RaNKEWEX9UdKbdHQdc3nu95K
> > > yg&u
> > > tm_medium=email>
> > > <https://substack.com/app-link/post?publication_id=283943&post_id=
> > > 1206 
> > > 01591&utm_source=substack&utm_medium=email&isFreemail=true&comment
> > > s=tr 
> > > ue&token=eyJ1c2VyX2lkIjo0NjU3NjIyNCwicG9zdF9pZCI6MTIwNjAxNTkxLCJpY
> > > XQiO 
> > > jE2ODQxMzczNDksImV4cCI6MTY4NjcyOTM0OSwiaXNzIjoicHViLTI4Mzk0MyIsInN
> > > 1YiI 
> > > 6InBvc3QtcmVhY3Rpb24ifQ.tQXs6dMHmFEgLKB4vUG4uGJTUSE0N7UdAH1qsj86mC
> > > A&ut m_source=substack&utm_medium=email>
> > > <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwc
> > > m9hY 
> > > 2guc3Vic3RhY2suY29tL3AvdGhlLWZ1dHVyZS1vZi1wNC1vbmUtcGVyc3BlY3RpdmU
> > > _dXR 
> > > tX3NvdXJjZT1zdWJzdGFjayZ1dG1fbWVkaXVtPWVtYWlsJmFjdGlvbj1yZXN0YWNrL
> > > WNvb 
> > > W1lbnQiLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzY
> > > yMjQ 
> > > sImlhdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsI
> > > nN1Y 
> > > iI6ImxpbmstcmVkaXJlY3QifQ.hfc7rMzN53dfpysbLYXxmtkw9siB9dPnC71KIavCOTo?
> > > &utm_source=substack&utm_medium=email>
> > > 
> > > Share
> > > <https://substack.com/app-link/post?publication_id=283943&post_id=
> > > 1206 
> > > 01591&utm_source=substack&utm_medium=email&utm_content=share&actio
> > > n=sh 
> > > are&triggerShare=true&isFreemail=true&token=eyJ1c2VyX2lkIjo0NjU3Nj
> > > IyNC
> > > wicG9zdF9pZCI6MTIwNjAxNTkxLCJpYXQiOjE2ODQxMzczNDksImV4cCI6MTY4Njcy
> > > OTM0 
> > > OSwiaXNzIjoicHViLTI4Mzk0MyIsInN1YiI6InBvc3QtcmVhY3Rpb24ifQ.tQXs6dM
> > > HmFE
> > > gLKB4vUG4uGJTUSE0N7UdAH1qsj86mCA>
> > > 
> > > 
> > > The P4 workshop has now been chaired by both co-founders of 
> > > Systems Approach, but this year the P4 landscape has shifted again 
> > > with Intel’s announcement that Tofino 3, its flagship P4-powered 
> > > switching chip, would not go ahead. There is much more to P4 than 
> > > Tofino, however, as we explore in this week’s newsletter.
> > > ------------------------------
> > > 
> > > The P4 Workshop
> > > <https://substack.com/redirect/4b87822a-70c6-4ffa-864a-45637b26909
> > > c?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > was a couple weeks ago, and as General Chair, I went into it with 
> > > a fair amount of trepidation. My concern was that Intel’s 
> > > announcement 
> > > <https://substack.com/redirect/a0831b60-fff8-4ded-88c7-f3e6ef7e1e5
> > > 4?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > earlier this year that they’re cancelling development of the 
> > > Tofino 3 switching chip would have a chilling effect, not only on 
> > > the Workshop, but also on the future of P4. That concern has been 
> > > voiced in several forums, including SIGCOMM’s Slack workspace 
> > > <https://substack.com/redirect/300dcab9-6056-4c82-9627-1e08ea449b0
> > > e?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>,
> > > with members of the P4 Advisory Board 
> > > <https://substack.com/redirect/2fcc130d-cf74-4d68-be4e-d531cbb379b
> > > 4?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > making reassuring pronouncements in various settings. (See for 
> > > example, Nick McKeown’s post to the P4 Forum 
> > > <https://substack.com/redirect/b0f9bc57-686d-4962-8676-c234ebcbd12
> > > f?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>,
> > > and Nick along with Nate Foster and Jennifer Rexford discussing 
> > > the future of Network Programmability on The Networking Channel 
> > > <https://substack.com/redirect/f2016908-8afb-4538-be3d-5d0e6aab0b3
> > > c?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > ).
> > > 
> > > I won’t try to give a point-by-point replay of what Nick, Nate, 
> > > and Jen and others have been saying, except to observe that at a 
> > > high level it can be summarized as follows:
> > > 
> > > *Programmable Networks  >>  P4 Language  >>  Tofino Switching 
> > > Chip*
> > > 
> > > They point out, for example, that Tofino is just one of many 
> > > interesting backend targets for P4 programs (SmartNICs 
> > > <https://substack.com/redirect/f5d04cdb-5915-42ab-9904-a20c38eeab3
> > > 3?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > and IPUs
> > > <https://substack.com/redirect/a0ddc803-4ea3-4248-96a1-bdc33f86cd1
> > > 5?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > being the next “big deal”) and P4 is one of many tools being used 
> > > to inject functionality into the end-to-end network path (DPDK and 
> > > eBPF being two active projects that people are integrating with P4).
> > > Ultimately, the value of programmability comes from having 
> > > visibility and control over the network, and there are many complementary approaches to making that happen.
> > > With that background, I do have three takeaways from what turned 
> > > out to be an interesting and vibrant two days at the P4 Workshop 
> > > (despite my initial concerns).
> > > 
> > > First, we’re often so focused on P4 as a tool to program the 
> > > forwarding pipeline that we forget the other half of its value
> > > proposition: It also provides a way to specify the behavior of a 
> > > pipeline (independent of how that pipeline is implemented). We 
> > > talk about this idea, and the value of being able to auto-generate 
> > > the Control API, in the P4 chapter of our SDN Book 
> > > <https://substack.com/redirect/4b87822a-70c6-4ffa-864a-45637b26909c?j=eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>.
> > > Rob Sherwood made a similar argument 
> > > <https://substack.com/redirect/2442a19b-b82c-410c-8692-f50b8973387
> > > 5?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > at the P4 Workshop. It is now becoming a reality as companies like 
> > > Google are starting to use such behavioral definitions as a 
> > > Hardware Abstraction Layer (see Parveen Patel’s Keynote 
> > > <https://substack.com/redirect/e656a164-3fef-497b-9628-ffeb700108a
> > > f?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > at the Workshop). This makes me hopeful that we are rapidly 
> > > approaching the day when a P4 program (plus the generated P4RT
> > > interface) will become the standard way network providers specify 
> > > their requirements to network vendors, and proposed new features 
> > > (whether proprietary or standard) will be specified by a P4 
> > > program (potentially augmenting the intuition and design rationale presented in an RFC).
> > > 
> > > *As an aside, I couldn’t help but notice the similarities between 
> > > the architecture Parveen described and the way P4 has been used to 
> > > program the forwarding plane of the 5G Mobile Core 
> > > <https://substack.com/redirect/64e4cac9-e6fd-4652-8e36-09f50ff1305c?j=eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>.
> > > Both include a P4-based “abstract forwarding model” that’s 
> > > independent of the underlying implementation details*.
> > > 
> > > Second, it is common to divide forwarding pipelines into “programmable”
> > > versus “fixed function”, but this glosses over what might be the 
> > > more important distinction: whether the pipeline is *open* or *closed*.
> > > Even “fixed function” pipelines are increasingly flexible–it’s 
> > > just a question of how restrictive the vendor is in who they allow 
> > > to make changes. This restriction may have the biggest impact on 
> > > researchers who want to experiment with a new feature (especially 
> > > ones that do not yet have a proven market), but maybe less so in 
> > > the commercial world where incentives to make changes are 
> > > (arguably) well-defined. Using P4 as the “spec language” (as I 
> > > just outlined) has the potential to accelerate the process on the 
> > > commercial side. On the research side, there is a strong argument 
> > > in favor of using Tofino 2 to demonstrate the feasibility and 
> > > value of new ideas (12.8 Tb/s still makes for a credible 
> > > Proof-of-Concept), and repeating the refrain yet again, P4-as-spec 
> > > makes for a compelling tech transfer story. If that were to 
> > > happen, it would be interesting to see how vendors and chip 
> > > designers adapt to reduce their spec-to-hardware implementation overhead. I would argue that programmable forwarding planes have a time-to-market advantage even for closed solutions.
> > > 
> > > Third, our focus on quantifiable metrics makes it easy to forget 
> > > about the less quantifiable aspects of programmability. At its 
> > > core, P4 is a programming language that does a good job of 
> > > abstracting the essence of a packet forwarding pipeline. It is 
> > > enormously impressive that a P4 program can be compiled onto a 
> > > PISA-based 
> > > <https://substack.com/redirect/01cfb256-04e7-49f8-9600-55e1471955b
> > > 8?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > switching chip that has the same performance, die area, cost, and 
> > > power consumption of a fixed-function ASIC (and that equivalency 
> > > was probably necessary for P4 to be taken seriously), but hitting 
> > > that quantifiable mark is not sufficient. Well-designed languages 
> > > are software tools that bring clarity to the intellectual 
> > > challenge of programming. For me, the biggest “aha” moment of the 
> > > Workshop was when Chris Sommers (long-time P4 contributor and new 
> > > co-Chair of the API Working Group) started rattling off all the 
> > > functions he’d been involved in writing in P4, and remarking on 
> > > how natural P4 makes that process. There is certainly room to add 
> > > new language features as P4 expands its domain to include 
> > > SmartNICs and IPUs—as Chris and the other WG chairs are now pursuing—but having an existing target to evolve is a great position to be in.
> > > 
> > > One common thread that weaves its way through these three 
> > > takeaways is that Intel’s cancellation of the Tofino 3 chip is a 
> > > potentially helpful forcing
> > > function: The P4 community has to demonstrate the value of the 
> > > language without being buttressed by ever-improving performance 
> > > numbers that have more to do with 7nm semiconductor technology 
> > > than anything networking people have done. I saw a lot of evidence 
> > > that exactly that is happening at last month’s workshop. The march 
> > > to programmable networks is inevitable (in my view), and I’m still 
> > > optimistic about the role P4 will play a central role.
> > > 
> > > Systems Approach is reader-supported and we are committed to 
> > > keeping our books and articles open to all. To receive new posts 
> > > and support our work, consider becoming a free or paid subscriber.
> > > 
> > > Upgrade to paid
> > > <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwc
> > > m9hY
> > > 2guc3Vic3RhY2suY29tL3N1YnNjcmliZT90b2tlbj1leUoxYzJWeVgybGtJam8wTmp
> > > VM05 
> > > qSXlOQ3dpYVdGMElqb3hOamcwTVRNM016UTVMQ0psZUhBaU9qRTJPRFkzTWprek5Ea
> > > 3NJb 
> > > Wx6Y3lJNkluQjFZaTB5T0RNNU5ETWlMQ0p6ZFdJaU9pSmphR1ZqYTI5MWRDSjkuLW5
> > > LYUM 
> > > tRE1weEw3Smp0bHh4U1I0aEFFLXQ2QWltZ0lENFlFRC1BbWJicyZ1dG1fc291cmNlP
> > > XBvc 
> > > 3QiLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzYyMjQ
> > > sIml 
> > > hdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsInN1Y
> > > iI6I 
> > > mxpbmstcmVkaXJlY3QifQ.lWjTeCLgxCt9OYaegbWWppVXvEzTeJBQs6psa45Oowg?
> > > &utm
> > > _medium=email&utm_source=subscribe-widget-preamble&utm_content=120
> > > 6015
> > > 91>
> > > 
> > > ------------------------------
> > > 
> > > We continue to run into people who want to translate our books 
> > > into other languages, and if you are one of them, you should 
> > > definitely reach out to us. The latest entrant is a Portuguese 
> > > translation 
> > > <https://substack.com/redirect/e0ad5652-867f-4d2c-8cd6-3fc8b0f8eda
> > > 0?j= 
> > > eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>
> > > of our Private 5G book by Edmar Candeia Gurjão. You can find other 
> > > translations of our books here 
> > > <https://substack.com/redirect/0d0dad50-a3db-42e9-9ad6-95318436a540?j=eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>.
> > > 
> > > 
> > > You can follow us on Mastodon
> > > <https://substack.com/redirect/3e6fc5cc-6a89-4a15-97bf-231cf12de9ec?j=eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>.
> > > 
> > > 
> > > You're currently a free subscriber to Systems Approach 
> > > <https://substack.com/redirect/3d94c242-4758-4bf3-9e1d-62b5ce3f23ab?j=eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>.
> > > For the full experience, upgrade your subscription.
> > > <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwc
> > > m9hY
> > > 2guc3Vic3RhY2suY29tL3N1YnNjcmliZT90b2tlbj1leUoxYzJWeVgybGtJam8wTmp
> > > VM05 
> > > qSXlOQ3dpYVdGMElqb3hOamcwTVRNM016UTVMQ0psZUhBaU9qRTJPRFkzTWprek5Ea
> > > 3NJb 
> > > Wx6Y3lJNkluQjFZaTB5T0RNNU5ETWlMQ0p6ZFdJaU9pSmphR1ZqYTI5MWRDSjkuLW5
> > > LYUM 
> > > tRE1weEw3Smp0bHh4U1I0aEFFLXQ2QWltZ0lENFlFRC1BbWJicyZ1dG1fc291cmNlP
> > > XBvc 
> > > 3QiLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzYyMjQ
> > > sIml 
> > > hdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsInN1Y
> > > iI6I 
> > > mxpbmstcmVkaXJlY3QifQ.lWjTeCLgxCt9OYaegbWWppVXvEzTeJBQs6psa45Oowg?
> > > &utm _source=substack&utm_medium=email&utm_content=postcta>
> > > 
> > > Upgrade to paid
> > > <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwc
> > > m9hY
> > > 2guc3Vic3RhY2suY29tL3N1YnNjcmliZT90b2tlbj1leUoxYzJWeVgybGtJam8wTmp
> > > VM05 
> > > qSXlOQ3dpYVdGMElqb3hOamcwTVRNM016UTVMQ0psZUhBaU9qRTJPRFkzTWprek5Ea
> > > 3NJb 
> > > Wx6Y3lJNkluQjFZaTB5T0RNNU5ETWlMQ0p6ZFdJaU9pSmphR1ZqYTI5MWRDSjkuLW5
> > > LYUM 
> > > tRE1weEw3Smp0bHh4U1I0aEFFLXQ2QWltZ0lENFlFRC1BbWJicyZ1dG1fc291cmNlP
> > > XBvc 
> > > 3QiLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzYyMjQ
> > > sIml 
> > > hdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsInN1Y
> > > iI6I 
> > > mxpbmstcmVkaXJlY3QifQ.lWjTeCLgxCt9OYaegbWWppVXvEzTeJBQs6psa45Oowg?
> > > &utm _source=substack&utm_medium=email&utm_content=postcta>
> > > 
> > > Like
> > > <https://substack.com/app-link/post?publication_id=283943&post_id=
> > > 1206
> > > 01591&utm_source=substack&isFreemail=true&submitLike=true&token=ey
> > > J1c2 
> > > VyX2lkIjo0NjU3NjIyNCwicG9zdF9pZCI6MTIwNjAxNTkxLCJyZWFjdGlvbiI6IuKd
> > > pCIs 
> > > ImlhdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMjgzOT
> > > QzIi 
> > > wic3ViIjoicmVhY3Rpb24ifQ.vN6AE7iLQvKkv98V0RaNKEWEX9UdKbdHQdc3nu95K
> > > yg&u
> > > tm_medium=email>
> > > Comment
> > > <https://substack.com/app-link/post?publication_id=283943&post_id=
> > > 1206 
> > > 01591&utm_source=substack&utm_medium=email&isFreemail=true&comment
> > > s=tr 
> > > ue&token=eyJ1c2VyX2lkIjo0NjU3NjIyNCwicG9zdF9pZCI6MTIwNjAxNTkxLCJpY
> > > XQiO 
> > > jE2ODQxMzczNDksImV4cCI6MTY4NjcyOTM0OSwiaXNzIjoicHViLTI4Mzk0MyIsInN
> > > 1YiI 
> > > 6InBvc3QtcmVhY3Rpb24ifQ.tQXs6dMHmFEgLKB4vUG4uGJTUSE0N7UdAH1qsj86mC
> > > A&ut m_source=substack&utm_medium=email>
> > > Restack
> > > <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwc
> > > m9hY 
> > > 2guc3Vic3RhY2suY29tL3AvdGhlLWZ1dHVyZS1vZi1wNC1vbmUtcGVyc3BlY3RpdmU
> > > _dXR 
> > > tX3NvdXJjZT1zdWJzdGFjayZ1dG1fbWVkaXVtPWVtYWlsJmFjdGlvbj1yZXN0YWNrL
> > > WNvb 
> > > W1lbnQiLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzY
> > > yMjQ 
> > > sImlhdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsI
> > > nN1Y 
> > > iI6ImxpbmstcmVkaXJlY3QifQ.hfc7rMzN53dfpysbLYXxmtkw9siB9dPnC71KIavCOTo?
> > > &utm_source=substack&utm_medium=email>
> > > 
> > > 
> > > © 2023 Systems Approach, LLC
> > > 548 Market Street PMB 72296, San Francisco, CA 94104 Unsubscribe 
> > > <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwc
> > > m9hY 
> > > 2guc3Vic3RhY2suY29tL2FjdGlvbi9kaXNhYmxlX2VtYWlsP3Rva2VuPWV5SjFjMlZ
> > > 5WDJ 
> > > sa0lqbzBOalUzTmpJeU5Dd2ljRzl6ZEY5cFpDSTZNVEl3TmpBeE5Ua3hMQ0pwWVhRa
> > > U9qR 
> > > TJPRFF4TXpjek5Ea3NJbVY0Y0NJNk1UWTROamN5T1RNME9Td2lhWE56SWpvaWNIVml
> > > MVEk 
> > > 0TXprME15SXNJbk4xWWlJNkltUnBjMkZpYkdWZlpXMWhhV3dpZlEuRXdaUkN3ZEdjd
> > > G8xd 
> > > FdwbmRVd3h2eXZKUk5pVmlOX2h1ZDBVNERDbVZYayZleHBpcmVzPTM2NWQiLCJwIjo
> > > xMjA 
> > > 2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzYyMjQsImlhdCI6MTY4N
> > > DEzN 
> > > zM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsInN1YiI6ImxpbmstcmV
> > > kaXJ lY3QifQ.l6lDN2unh8JkUjeYEXjKiNFu7akDZ_FyOQUxaYockVM?>
> > > 
> > > [image: Start writing]
> > > <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zdWJzdGFjay5jb
> > > 20vc 
> > > 2lnbnVwP3V0bV9zb3VyY2U9c3Vic3RhY2smdXRtX21lZGl1bT1lbWFpbCZ1dG1fY29
> > > udGV 
> > > udD1mb290ZXImdXRtX2NhbXBhaWduPWF1dG9maWxsZWQtZm9vdGVyJmZyZWVTaWdud
> > > XBFb 
> > > WFpbD1tYXJpZUBtam1vbnRwZXRpdC5jb20iLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk
> > > 0Myw 
> > > iZiI6dHJ1ZSwidSI6NDY1NzYyMjQsImlhdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2N
> > > zI5M
> > > zQ5LCJpc3MiOiJwdWItMCIsInN1YiI6ImxpbmstcmVkaXJlY3QifQ.GtVGEVhyYZ__
> > > pv96
> > > bIPsssoL78rJrnvkIgcdwC4WQfs?>
> > > --
> > > Coin mailing list
> > > [e-mail]
> > > https://www.irtf.org/mailman/listinfo/coin
> > 
> > 

--
---
[e-mail]