Re: [Coin] Fwd: The Future of P4, Revisited

[[e-mail]]

+1

 

From: Coin <[e-mail]> On Behalf Of Marie-Jose Montpetit
Sent: Monday, May 15, 2023 2:41 PM
To: [e-mail]; Toerless Eckert <[e-mail]>
Cc: coin <[e-mail]>; [e-mail]
Subject: Re: [Coin] Fwd: The Future of P4, Revisited

 

Well I myself think the P4 implementation on RaspberryPI is a very good way for at least students to get initiated into COIN-type networking.

 

And the idea at the ONF P4 meeting to use it as a hardware abstraction is also worth looking into.

 

mjm

 

Marie-José Montpetit, Ph.D.

[e-mail] <mailto:[e-mail]> 

 

 


From: [e-mail] <mailto:[e-mail]>   <mailto:[e-mail]> <[e-mail]>
Reply: [e-mail] <mailto:[e-mail]>   <mailto:[e-mail]> <[e-mail]>
Date: May 15, 2023 at 2:04:47 PM
To: Toerless Eckert  <mailto:[e-mail]> <[e-mail]>, Marie-Jose Montpetit  <mailto:[e-mail]> <[e-mail]>
Cc: coin  <mailto:[e-mail]> <[e-mail]>, [e-mail] <mailto:[e-mail]>   <mailto:[e-mail]> <[e-mail]>
Subject:  RE: [Coin] Fwd: The Future of P4, Revisited 





Toerless, 

This is the latest network processor which supports use in a switch or nic. 

https://www.marvell.com/content/dam/marvell/en/company/media-kit/octeon-10/marvell-octeon-10-media-deck.pdf 

This is your Tofino replacement with hardware available from Marvell and P4 compiler and tools chain available from my company. 

Hemant 


-----Original Message----- 
From: Coin <[e-mail] <mailto:[e-mail]> > On Behalf Of Toerless Eckert 
Sent: Monday, May 15, 2023 1:04 PM 
To: Marie-Jose Montpetit <[e-mail] <mailto:[e-mail]> > 
Cc: coin <[e-mail] <mailto:[e-mail]> >; [e-mail] <mailto:[e-mail]>  
Subject: Re: [Coin] Fwd: The Future of P4, Revisited 

Thanks, Marie-Jose 

IMHO: 

- P4 in research short-term: 
Researchers can not do real-world relevant PoC with just BMV2, because it has no constraints. 
All those papers claiming some problem can be solved with P4 and then just use BMV2 are just 
so painful to read/listen-to. We really need a BMV2 with the exact constraints of Tofino, 
and given how Intel is deinvesting from Tofino, maybe they would finally be willing to 
publish those constraints outside of NDA so this can be implemented into BMV2. 

- SmartNICs: 
P4 is not the right DSL for SmartNICs even if Intel may say so because they want to 
sell SmartNICs and have a brand with P4. SmartNICs are much more flexible, and programming then 
in P4 constraints you. If you don't know anything better, try eBPF. That is AFAIK the 
most widely adopted DSL today to span general purpose CPU (host/router CPU), and SmartNICs. 

SmartNICs are a great target FPE for forwarding plane features you do not need on every hop, 
but for example only on few "can be more expensive" hops. WAN interface of site-edge-routers 
(Campus, Home, WAN) for example. But do not try to assume they would be used beyond that 
(on every hop == especially on higher aggregation speed links!). 

- P4/Coin for higher-than-forwarding-plane features: 
With Tofino disappearing, i think we should caution researchers to NOT invest cycles trying to 
implement solutions on P4 with the explicit purpose of "abusing" Tofino, aka: Program a 
non-forwarding-plane problem solution in P4 in the expectation that a fast P4 exeuction 
product like Tofino would be a great alternative/competition to other execution platforms 
(SmartNIC, CPU,...). We had several of those great ideas presented/shown in Coin, they 
where all about higher than forwarding-plane "compute" problems. But very quickly, with 
Tofino aging, i am sure CPU or SmartNIC slutions for the same problems can easily be 
shown to be more cost-effective. 

- Long-term forwarding plane research: 
The rejection of the industry (outside Intel) to support P4 for researchers on their router/switch 
forwarding planes (for a decade now) should primarily be a trigger for research funding 
such as from NSF/EU to better enable long-term forwarding plane research. IMHO there 
should simply be a single well funded effort for a Network Programming Element (NPE) equivalent 
to RISC-V. Aka: fully open source architecture and instruction set for a research NPE, 
competitive to industry products. There are lot of other benefits from such an effort 
that could help make that happen. 

Cheers 
Toerless 

On Mon, May 15, 2023 at 03:33:42AM -0700, Marie-Jose Montpetit wrote: 
> Interesting article for out community. 
> 
> mjm 
> 
> Marie-José Montpetit, Ph.D. 
> [e-mail] <mailto:[e-mail]>  
> 
> 
> 
> From: Larry Peterson from Systems Approach 
> <[e-mail] <mailto:[e-mail]> > <[e-mail] <mailto:[e-mail]> > 
> Reply: Larry Peterson from Systems Approach 
> <reply+1zsws7&rqag0&&3ca860fd966ac1a0bf575ed48379b6645b7df5edc0903e4c5 
> [e-mail] <mailto:[e-mail]> > 
> <reply+1zsws7&rqag0&&3ca860fd966ac1a0bf575ed48379b6645b7df5edc0903e4c5 
> [e-mail] <mailto:[e-mail]> > 
> Date: May 15, 2023 at 3:55:52 AM 
> To: [e-mail] <mailto:[e-mail]>  <[e-mail] <mailto:[e-mail]> > 
> <[e-mail] <mailto:[e-mail]> > 
> Subject: The Future of P4, Revisited 
> 
> The P4 Workshop was a couple weeks ago, and as General Chair, I went 
> into it with a fair amount of trepidation. My concern was that Intel’s 
> announcement earlier this year that they’re cancelling development of 
> the Tofino 3 switching chip would have a chilling effect, not only on 
> the Workshop, but also on the future of P4. That concern has been 
> voiced in several forums ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 
> ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 
> ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 
> ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 
> ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 
> ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ 
> ‌ ‌ ‌ Open in app 
> <https://open.substack.com/pub/systemsapproach/p/the-future-of-p4-one- 
> perspective?utm_source=email&redirect=app-store> 
> or online 
> <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwcm9hY 
> 2guc3Vic3RhY2suY29tL3AvdGhlLWZ1dHVyZS1vZi1wNC1vbmUtcGVyc3BlY3RpdmU_dG9 
> rZW49ZXlKMWMyVnlYMmxrSWpvME5qVTNOakl5TkN3aWNHOXpkRjlwWkNJNk1USXdOakF4T 
> lRreExDSnBZWFFpT2pFMk9EUXhNemN6TkRrc0ltVjRjQ0k2TVRZNE5qY3lPVE0wT1N3aWF 
> YTnpJam9pY0hWaUxUSTRNemswTXlJc0luTjFZaUk2SW5CdmMzUXRjbVZoWTNScGIyNGlmU 
> S50UVhzNmRNSG1GRWdMS0I0dlVHNHVHSlRVU0UwTjdVZEFIMXFzajg2bUNBIiwicCI6MTI 
> wNjAxNTkxLCJzIjoyODM5NDMsImYiOnRydWUsInUiOjQ2NTc2MjI0LCJpYXQiOjE2ODQxM 
> zczNDksImV4cCI6MTY4NjcyOTM0OSwiaXNzIjoicHViLTAiLCJzdWIiOiJsaW5rLXJlZGl 
> yZWN0In0.6L7-AmSfpdLfVXnKIHzqDqYX9u_tCmRVPJC2N9oP8gU?> 
> The Future of P4, Revisited 
> <https://substack.com/app-link/post?publication_id=283943 <https://substack.com/app-link/post?publication_id=283943&post_id=1206> &post_id=1206 
> 01591&utm_source=post-email-title&isFreemail=true&token=eyJ1c2VyX2lkIj 
> o0NjU3NjIyNCwicG9zdF9pZCI6MTIwNjAxNTkxLCJpYXQiOjE2ODQxMzczNDksImV4cCI6 
> MTY4NjcyOTM0OSwiaXNzIjoicHViLTI4Mzk0MyIsInN1YiI6InBvc3QtcmVhY3Rpb24ifQ 
> .tQXs6dMHmFEgLKB4vUG4uGJTUSE0N7UdAH1qsj86mCA> 
> 
> Larry Peterson 
> <https://substack.com/redirect/ff29b5a3-7ea9-4b83-b888-b0c517b2bedd?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> May 15 
> <https://substack.com/redirect/ff29b5a3-7ea9-4b83-b888-b0c517b2bedd?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> 
> <https://substack.com/app-link/post?publication_id=283943 <https://substack.com/app-link/post?publication_id=283943&post_id=1206> &post_id=1206 
> 01591&utm_source=substack&isFreemail=true&submitLike=true&token=eyJ1c2 
> VyX2lkIjo0NjU3NjIyNCwicG9zdF9pZCI6MTIwNjAxNTkxLCJyZWFjdGlvbiI6IuKdpCIs 
> ImlhdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMjgzOTQzIi 
> wic3ViIjoicmVhY3Rpb24ifQ.vN6AE7iLQvKkv98V0RaNKEWEX9UdKbdHQdc3nu95Kyg&u 
> tm_medium=email> 
> <https://substack.com/app-link/post?publication_id=283943 <https://substack.com/app-link/post?publication_id=283943&post_id=1206> &post_id=1206 
> 01591&utm_source=substack&utm_medium=email&isFreemail=true&comments=tr 
> ue&token=eyJ1c2VyX2lkIjo0NjU3NjIyNCwicG9zdF9pZCI6MTIwNjAxNTkxLCJpYXQiO 
> jE2ODQxMzczNDksImV4cCI6MTY4NjcyOTM0OSwiaXNzIjoicHViLTI4Mzk0MyIsInN1YiI 
> 6InBvc3QtcmVhY3Rpb24ifQ.tQXs6dMHmFEgLKB4vUG4uGJTUSE0N7UdAH1qsj86mCA&ut 
> m_source=substack&utm_medium=email> 
> <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwcm9hY 
> 2guc3Vic3RhY2suY29tL3AvdGhlLWZ1dHVyZS1vZi1wNC1vbmUtcGVyc3BlY3RpdmU_dXR 
> tX3NvdXJjZT1zdWJzdGFjayZ1dG1fbWVkaXVtPWVtYWlsJmFjdGlvbj1yZXN0YWNrLWNvb 
> W1lbnQiLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzYyMjQ 
> sImlhdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsInN1Y 
> iI6ImxpbmstcmVkaXJlY3QifQ.hfc7rMzN53dfpysbLYXxmtkw9siB9dPnC71KIavCOTo? 
> &utm_source=substack&utm_medium=email> 
> 
> Share 
> <https://substack.com/app-link/post?publication_id=283943 <https://substack.com/app-link/post?publication_id=283943&post_id=1206> &post_id=1206 
> 01591&utm_source=substack&utm_medium=email&utm_content=share&action=sh 
> are&triggerShare=true&isFreemail=true&token=eyJ1c2VyX2lkIjo0NjU3NjIyNC 
> wicG9zdF9pZCI6MTIwNjAxNTkxLCJpYXQiOjE2ODQxMzczNDksImV4cCI6MTY4NjcyOTM0 
> OSwiaXNzIjoicHViLTI4Mzk0MyIsInN1YiI6InBvc3QtcmVhY3Rpb24ifQ.tQXs6dMHmFE 
> gLKB4vUG4uGJTUSE0N7UdAH1qsj86mCA> 
> 
> 
> The P4 workshop has now been chaired by both co-founders of Systems 
> Approach, but this year the P4 landscape has shifted again with 
> Intel’s announcement that Tofino 3, its flagship P4-powered switching 
> chip, would not go ahead. There is much more to P4 than Tofino, 
> however, as we explore in this week’s newsletter. 
> ------------------------------ 
> 
> The P4 Workshop 
> <https://substack.com/redirect/4b87822a-70c6-4ffa-864a-45637b26909c?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> was a couple weeks ago, and as General Chair, I went into it with a 
> fair amount of trepidation. My concern was that Intel’s announcement 
> <https://substack.com/redirect/a0831b60-fff8-4ded-88c7-f3e6ef7e1e54?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> earlier this year that they’re cancelling development of the Tofino 3 
> switching chip would have a chilling effect, not only on the Workshop, 
> but also on the future of P4. That concern has been voiced in several 
> forums, including SIGCOMM’s Slack workspace 
> <https://substack.com/redirect/300dcab9-6056-4c82-9627-1e08ea449b0e?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>, 
> with members of the P4 Advisory Board 
> <https://substack.com/redirect/2fcc130d-cf74-4d68-be4e-d531cbb379b4?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> making reassuring pronouncements in various settings. (See for 
> example, Nick McKeown’s post to the P4 Forum 
> <https://substack.com/redirect/b0f9bc57-686d-4962-8676-c234ebcbd12f?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>, 
> and Nick along with Nate Foster and Jennifer Rexford discussing the 
> future of Network Programmability on The Networking Channel 
> <https://substack.com/redirect/f2016908-8afb-4538-be3d-5d0e6aab0b3c?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> ). 
> 
> I won’t try to give a point-by-point replay of what Nick, Nate, and 
> Jen and others have been saying, except to observe that at a high 
> level it can be summarized as follows: 
> 
> *Programmable Networks >> P4 Language >> Tofino Switching Chip* 
> 
> They point out, for example, that Tofino is just one of many 
> interesting backend targets for P4 programs (SmartNICs 
> <https://substack.com/redirect/f5d04cdb-5915-42ab-9904-a20c38eeab33?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> and IPUs 
> <https://substack.com/redirect/a0ddc803-4ea3-4248-96a1-bdc33f86cd15?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> being the next “big deal”) and P4 is one of many tools being used to 
> inject functionality into the end-to-end network path (DPDK and eBPF 
> being two active projects that people are integrating with P4). 
> Ultimately, the value of programmability comes from having visibility 
> and control over the network, and there are many complementary approaches to making that happen. 
> With that background, I do have three takeaways from what turned out 
> to be an interesting and vibrant two days at the P4 Workshop (despite 
> my initial concerns). 
> 
> First, we’re often so focused on P4 as a tool to program the 
> forwarding pipeline that we forget the other half of its value 
> proposition: It also provides a way to specify the behavior of a 
> pipeline (independent of how that pipeline is implemented). We talk 
> about this idea, and the value of being able to auto-generate the 
> Control API, in the P4 chapter of our SDN Book 
> <https://substack.com/redirect/4b87822a-70c6-4ffa-864a-45637b26909c?j=eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>. 
> Rob Sherwood made a similar argument 
> <https://substack.com/redirect/2442a19b-b82c-410c-8692-f50b89733875?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> at the P4 Workshop. It is now becoming a reality as companies like 
> Google are starting to use such behavioral definitions as a Hardware 
> Abstraction Layer (see Parveen Patel’s Keynote 
> <https://substack.com/redirect/e656a164-3fef-497b-9628-ffeb700108af?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> at the Workshop). This makes me hopeful that we are rapidly 
> approaching the day when a P4 program (plus the generated P4RT 
> interface) will become the standard way network providers specify 
> their requirements to network vendors, and proposed new features 
> (whether proprietary or standard) will be specified by a P4 program 
> (potentially augmenting the intuition and design rationale presented in an RFC). 
> 
> *As an aside, I couldn’t help but notice the similarities between the 
> architecture Parveen described and the way P4 has been used to program 
> the forwarding plane of the 5G Mobile Core 
> <https://substack.com/redirect/64e4cac9-e6fd-4652-8e36-09f50ff1305c?j=eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>. 
> Both include a P4-based “abstract forwarding model” that’s independent 
> of the underlying implementation details*. 
> 
> Second, it is common to divide forwarding pipelines into “programmable” 
> versus “fixed function”, but this glosses over what might be the more 
> important distinction: whether the pipeline is *open* or *closed*. 
> Even “fixed function” pipelines are increasingly flexible–it’s just a 
> question of how restrictive the vendor is in who they allow to make 
> changes. This restriction may have the biggest impact on researchers 
> who want to experiment with a new feature (especially ones that do not 
> yet have a proven market), but maybe less so in the commercial world 
> where incentives to make changes are (arguably) well-defined. Using P4 
> as the “spec language” (as I just outlined) has the potential to 
> accelerate the process on the commercial side. On the research side, 
> there is a strong argument in favor of using Tofino 2 to demonstrate 
> the feasibility and value of new ideas (12.8 Tb/s still makes for a 
> credible Proof-of-Concept), and repeating the refrain yet again, 
> P4-as-spec makes for a compelling tech transfer story. If that were to 
> happen, it would be interesting to see how vendors and chip designers 
> adapt to reduce their spec-to-hardware implementation overhead. I 
> would argue that programmable forwarding planes have a time-to-market advantage even for closed solutions. 
> 
> Third, our focus on quantifiable metrics makes it easy to forget about 
> the less quantifiable aspects of programmability. At its core, P4 is a 
> programming language that does a good job of abstracting the essence 
> of a packet forwarding pipeline. It is enormously impressive that a P4 
> program can be compiled onto a PISA-based 
> <https://substack.com/redirect/01cfb256-04e7-49f8-9600-55e1471955b8?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> switching chip that has the same performance, die area, cost, and 
> power consumption of a fixed-function ASIC (and that equivalency was 
> probably necessary for P4 to be taken seriously), but hitting that 
> quantifiable mark is not sufficient. Well-designed languages are 
> software tools that bring clarity to the intellectual challenge of 
> programming. For me, the biggest “aha” moment of the Workshop was when 
> Chris Sommers (long-time P4 contributor and new co-Chair of the API 
> Working Group) started rattling off all the functions he’d been 
> involved in writing in P4, and remarking on how natural P4 makes that 
> process. There is certainly room to add new language features as P4 
> expands its domain to include SmartNICs and IPUs—as Chris and the 
> other WG chairs are now pursuing—but having an existing target to evolve is a great position to be in. 
> 
> One common thread that weaves its way through these three takeaways is 
> that Intel’s cancellation of the Tofino 3 chip is a potentially 
> helpful forcing 
> function: The P4 community has to demonstrate the value of the 
> language without being buttressed by ever-improving performance 
> numbers that have more to do with 7nm semiconductor technology than 
> anything networking people have done. I saw a lot of evidence that 
> exactly that is happening at last month’s workshop. The march to 
> programmable networks is inevitable (in my view), and I’m still 
> optimistic about the role P4 will play a central role. 
> 
> Systems Approach is reader-supported and we are committed to keeping 
> our books and articles open to all. To receive new posts and support 
> our work, consider becoming a free or paid subscriber. 
> 
> Upgrade to paid 
> <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwcm9hY 
> 2guc3Vic3RhY2suY29tL3N1YnNjcmliZT90b2tlbj1leUoxYzJWeVgybGtJam8wTmpVM05 
> qSXlOQ3dpYVdGMElqb3hOamcwTVRNM016UTVMQ0psZUhBaU9qRTJPRFkzTWprek5Ea3NJb 
> Wx6Y3lJNkluQjFZaTB5T0RNNU5ETWlMQ0p6ZFdJaU9pSmphR1ZqYTI5MWRDSjkuLW5LYUM 
> tRE1weEw3Smp0bHh4U1I0aEFFLXQ2QWltZ0lENFlFRC1BbWJicyZ1dG1fc291cmNlPXBvc 
> 3QiLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzYyMjQsIml 
> hdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsInN1YiI6I 
> mxpbmstcmVkaXJlY3QifQ.lWjTeCLgxCt9OYaegbWWppVXvEzTeJBQs6psa45Oowg?&utm 
> _medium=email&utm_source=subscribe-widget-preamble&utm_content=1206015 
> 91> 
> 
> ------------------------------ 
> 
> We continue to run into people who want to translate our books into 
> other languages, and if you are one of them, you should definitely 
> reach out to us. The latest entrant is a Portuguese translation 
> <https://substack.com/redirect/e0ad5652-867f-4d2c-8cd6-3fc8b0f8eda0?j= 
> eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0> 
> of our Private 5G book by Edmar Candeia Gurjão. You can find other 
> translations of our books here 
> <https://substack.com/redirect/0d0dad50-a3db-42e9-9ad6-95318436a540?j=eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>. 
> 
> 
> You can follow us on Mastodon 
> <https://substack.com/redirect/3e6fc5cc-6a89-4a15-97bf-231cf12de9ec?j=eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>. 
> 
> 
> You're currently a free subscriber to Systems Approach 
> <https://substack.com/redirect/3d94c242-4758-4bf3-9e1d-62b5ce3f23ab?j=eyJ1IjoicnFhZzAifQ.CwdsLjQyIcRXfYRGUucLsXfrvggM3KKa9Z1jxtXNWH0>. 
> For the full experience, upgrade your subscription. 
> <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwcm9hY 
> 2guc3Vic3RhY2suY29tL3N1YnNjcmliZT90b2tlbj1leUoxYzJWeVgybGtJam8wTmpVM05 
> qSXlOQ3dpYVdGMElqb3hOamcwTVRNM016UTVMQ0psZUhBaU9qRTJPRFkzTWprek5Ea3NJb 
> Wx6Y3lJNkluQjFZaTB5T0RNNU5ETWlMQ0p6ZFdJaU9pSmphR1ZqYTI5MWRDSjkuLW5LYUM 
> tRE1weEw3Smp0bHh4U1I0aEFFLXQ2QWltZ0lENFlFRC1BbWJicyZ1dG1fc291cmNlPXBvc 
> 3QiLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzYyMjQsIml 
> hdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsInN1YiI6I 
> mxpbmstcmVkaXJlY3QifQ.lWjTeCLgxCt9OYaegbWWppVXvEzTeJBQs6psa45Oowg?&utm 
> _source=substack&utm_medium=email&utm_content=postcta> 
> 
> Upgrade to paid 
> <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwcm9hY 
> 2guc3Vic3RhY2suY29tL3N1YnNjcmliZT90b2tlbj1leUoxYzJWeVgybGtJam8wTmpVM05 
> qSXlOQ3dpYVdGMElqb3hOamcwTVRNM016UTVMQ0psZUhBaU9qRTJPRFkzTWprek5Ea3NJb 
> Wx6Y3lJNkluQjFZaTB5T0RNNU5ETWlMQ0p6ZFdJaU9pSmphR1ZqYTI5MWRDSjkuLW5LYUM 
> tRE1weEw3Smp0bHh4U1I0aEFFLXQ2QWltZ0lENFlFRC1BbWJicyZ1dG1fc291cmNlPXBvc 
> 3QiLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzYyMjQsIml 
> hdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsInN1YiI6I 
> mxpbmstcmVkaXJlY3QifQ.lWjTeCLgxCt9OYaegbWWppVXvEzTeJBQs6psa45Oowg?&utm 
> _source=substack&utm_medium=email&utm_content=postcta> 
> 
> Like 
> <https://substack.com/app-link/post?publication_id=283943 <https://substack.com/app-link/post?publication_id=283943&post_id=1206> &post_id=1206 
> 01591&utm_source=substack&isFreemail=true&submitLike=true&token=eyJ1c2 
> VyX2lkIjo0NjU3NjIyNCwicG9zdF9pZCI6MTIwNjAxNTkxLCJyZWFjdGlvbiI6IuKdpCIs 
> ImlhdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMjgzOTQzIi 
> wic3ViIjoicmVhY3Rpb24ifQ.vN6AE7iLQvKkv98V0RaNKEWEX9UdKbdHQdc3nu95Kyg&u 
> tm_medium=email> 
> Comment 
> <https://substack.com/app-link/post?publication_id=283943 <https://substack.com/app-link/post?publication_id=283943&post_id=1206> &post_id=1206 
> 01591&utm_source=substack&utm_medium=email&isFreemail=true&comments=tr 
> ue&token=eyJ1c2VyX2lkIjo0NjU3NjIyNCwicG9zdF9pZCI6MTIwNjAxNTkxLCJpYXQiO 
> jE2ODQxMzczNDksImV4cCI6MTY4NjcyOTM0OSwiaXNzIjoicHViLTI4Mzk0MyIsInN1YiI 
> 6InBvc3QtcmVhY3Rpb24ifQ.tQXs6dMHmFEgLKB4vUG4uGJTUSE0N7UdAH1qsj86mCA&ut 
> m_source=substack&utm_medium=email> 
> Restack 
> <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwcm9hY 
> 2guc3Vic3RhY2suY29tL3AvdGhlLWZ1dHVyZS1vZi1wNC1vbmUtcGVyc3BlY3RpdmU_dXR 
> tX3NvdXJjZT1zdWJzdGFjayZ1dG1fbWVkaXVtPWVtYWlsJmFjdGlvbj1yZXN0YWNrLWNvb 
> W1lbnQiLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzYyMjQ 
> sImlhdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsInN1Y 
> iI6ImxpbmstcmVkaXJlY3QifQ.hfc7rMzN53dfpysbLYXxmtkw9siB9dPnC71KIavCOTo? 
> &utm_source=substack&utm_medium=email> 
> 
> 
> © 2023 Systems Approach, LLC 
> 548 Market Street PMB 72296, San Francisco, CA 94104 Unsubscribe 
> <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zeXN0ZW1zYXBwcm9hY 
> 2guc3Vic3RhY2suY29tL2FjdGlvbi9kaXNhYmxlX2VtYWlsP3Rva2VuPWV5SjFjMlZ5WDJ 
> sa0lqbzBOalUzTmpJeU5Dd2ljRzl6ZEY5cFpDSTZNVEl3TmpBeE5Ua3hMQ0pwWVhRaU9qR 
> TJPRFF4TXpjek5Ea3NJbVY0Y0NJNk1UWTROamN5T1RNME9Td2lhWE56SWpvaWNIVmlMVEk 
> 0TXprME15SXNJbk4xWWlJNkltUnBjMkZpYkdWZlpXMWhhV3dpZlEuRXdaUkN3ZEdjdG8xd 
> FdwbmRVd3h2eXZKUk5pVmlOX2h1ZDBVNERDbVZYayZleHBpcmVzPTM2NWQiLCJwIjoxMjA 
> 2MDE1OTEsInMiOjI4Mzk0MywiZiI6dHJ1ZSwidSI6NDY1NzYyMjQsImlhdCI6MTY4NDEzN 
> zM0OSwiZXhwIjoxNjg2NzI5MzQ5LCJpc3MiOiJwdWItMCIsInN1YiI6ImxpbmstcmVkaXJ 
> lY3QifQ.l6lDN2unh8JkUjeYEXjKiNFu7akDZ_FyOQUxaYockVM?> 
> 
> [image: Start writing] 
> <https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9zdWJzdGFjay5jb20vc 
> 2lnbnVwP3V0bV9zb3VyY2U9c3Vic3RhY2smdXRtX21lZGl1bT1lbWFpbCZ1dG1fY29udGV 
> udD1mb290ZXImdXRtX2NhbXBhaWduPWF1dG9maWxsZWQtZm9vdGVyJmZyZWVTaWdudXBFb 
> WFpbD1tYXJpZUBtam1vbnRwZXRpdC5jb20iLCJwIjoxMjA2MDE1OTEsInMiOjI4Mzk0Myw 
> iZiI6dHJ1ZSwidSI6NDY1NzYyMjQsImlhdCI6MTY4NDEzNzM0OSwiZXhwIjoxNjg2NzI5M 
> zQ5LCJpc3MiOiJwdWItMCIsInN1YiI6ImxpbmstcmVkaXJlY3QifQ.GtVGEVhyYZ__pv96 
> bIPsssoL78rJrnvkIgcdwC4WQfs?> 

> -- 
> Coin mailing list 
> [e-mail] <mailto:[e-mail]>  
> https://www.irtf.org/mailman/listinfo/coin 


-- 
--- 
[e-mail] <mailto:[e-mail]>  

-- 
Coin mailing list 
[e-mail] <mailto:[e-mail]>  
https://www.irtf.org/mailman/listinfo/coin