Re: [Apn] APN Charter revision

[David Lake <[e-mail]>]

Ted

Thank you – I will take this discussion to that email list but I do want to specifically answer one of your points first and I apologise to the group for the long (and top-posted) reply but I think we have hit on some very fundamental areas which APN points to.

WebRTC is only PART of the solution because it looks at the application alone.   Yes, there is signalling of all aspects in there as there is in VoLTE (actually the same signalling), but the art of providing a good quality of outcome is about the co-ordination of the resources and the requirements.

I don’t accept that the system is ‘too complex’ – we design networks and we know the speeds, feeds, operation and capacity of every single part of that design.  Even in a poorly-managed transport network such as a road system, we know, can model and can predict outcomes.  In a well-regulated system (power, gas, railways, etc) those outcomes are completely deterministic.  If we break the problem-space down to small domains, within and across domains we should be able to model and describe the outcome of the networks on a inter and intra-domain basis.

I know this is the case – I spent many (many) years deploying Enterprise IP Telephony [1].  We understood to the bit what the application needed in terms of RTT, bandwidth, packetization and jitter on a per-codec basis.   We knew what the Layer 2 queuing would need to be, how to map the traffic to Layer 3 classes of service and what congestion management would need to be in place for each path.  If that path was not available, we either instructed the application to downrate the codec (SDP negotiation) or denied the session not just because allowing it would be detrimental to the requested session but it would potentially impact the service level of existing sessions.

But the way we did that was mainly through static configuration – many nights spent counting required G.711 or G.729 sessions and working out what the peak call loading would be to ensure that enough EF-class traffic was available in the MPLS classes.  The best way to do this was a piece of A0 paper and a pencil…

Now, what SDN has given us is the promise of being able to dynamically program the underlying infrastructure based on external factors and yes, that could be the SIP signalling in WebRTC because that gives us the chance to deny/negotiate/allow the application class in tandem with the network programming – the ability for me as a consumer to put value on the service I want.  My own research is focusing on how we can steer traffic through UPFs with different performances based on more granular metadata than the TEID (which is all we have to go on at the moment unless I have a separate, parallel programming API).

I also don’t think the service DOES work today.  For those of us in the UK, BBC Radio 4’s “Today” programme is currently compelling morning listening (unfortunately…)  One aspect of the programme is the remote interviews with politicians.  The way this used to be done was over ISDN2 lines or a dedicated radio car (actually a converted London Black Cab with a heap of antennae).  The quality of the outside broadcasts was quite astonishing – crisp, clear, consistent throughput.  It was a precisely engineered system introduced in the mid 1970s.   Now, many of the interviews are done over Internet links and the quality has dropped to the point where it is normal to hear codec renegotiations mid-stream, sessions dropping out, lost packets, long latency.   Yes, this is usually over ‘consumer’ broadband or 4G but why can’t a 120kbit/s application request a specific outcome for a duration of time given that ostensibly we have FAR more end-point bandwidth than is actually needed?   WebRTC can only do that if the session requirements at an application level are mirrored in provisioning in the network.

With apologies for the long post which I will now take to the architecture list although I suspect that IRTF may be a better home...

David

[1] I was actually part of a team that ‘rescued’ poor IP Telephony installations.   90% of the issues we hit were not due to the application but the way that QoS and mapping of Call Admission Control to network queues, especially MPLS classes on WAN links, had not been carried out.   Bandwidth, especially an MPLS EF-class, is very expensive so the key was to be able to balance application requirements to available resources in an economically-viable manner.  The same is true today as we move to regulated, expensive RF spectrum as our predominant last-mile access method.

From: Ted Hardie <[e-mail]>
Date: Wednesday, 19 October 2022 at 09:33
To: Lake, David (PG/R - Comp Sci & Elec Eng) <[e-mail]>
Cc: Sankar Ramamoorthi <[e-mail]>, Donald Eastlake <[e-mail]>, Andrew Alston - IETF <[e-mail]>, [e-mail] <[e-mail]>
Subject: Re: [Apn] APN Charter revision
Top-posting to answer David's question of "if APN is not the place to discuss this, where should it go?" I think the answer is probably [e-mail]<mailto:[e-mail]>, which is the public list for the IAB and IETF discussions of Internet architecture questions.  I'm not sure how many folks on that list are currently tracking APN, though, so you will probably need to articulate the question independently of this discussion.

regards,

Ted

On Tue, Oct 18, 2022 at 9:05 PM David Lake <[e-mail]<mailto:[e-mail]>> wrote:
<dl> In-line again (and thank you for the offer of tea which I will, of course, graciously accept 😊  because it sounds like it could be a great conversation… ) </dl>

From: Ted Hardie <[e-mail]<mailto:[e-mail]>>
Date: Tuesday, 18 October 2022 at 12:11
To: Lake, David (PG/R - Comp Sci & Elec Eng) <[e-mail]<mailto:[e-mail]>>
Cc: Sankar Ramamoorthi <[e-mail]<mailto:[e-mail]>>, Donald Eastlake <[e-mail]<mailto:[e-mail]>>, Andrew Alston - IETF <[e-mail]>, [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
Subject: Re: [Apn] APN Charter revision
Responses in-line.

On Tue, Oct 18, 2022 at 10:45 AM David Lake <[e-mail]<mailto:[e-mail]>> wrote:
<dl> in-line </dl>

From: Ted Hardie <[e-mail]<mailto:[e-mail]>>
Date: Tuesday, 18 October 2022 at 10:12
To: Lake, David (PG/R - Comp Sci & Elec Eng) <[e-mail]<mailto:[e-mail]>>
Cc: Sankar Ramamoorthi <[e-mail]<mailto:[e-mail]>>, Donald Eastlake <[e-mail]<mailto:[e-mail]>>, Andrew Alston - IETF <[e-mail]>, [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
Subject: Re: [Apn] APN Charter revision
Some thoughts in-line

On Tue, Oct 18, 2022 at 8:44 AM David Lake <d.lake=[e-mail]<mailto:[e-mail]>> wrote:
Sankar

I completely agree with you.

Operators already have the ability to tweak data flows within their domain but what we are missing is any sense that when I pay my money for a service ALL the operators and providers in the supply chain will provide against my order.

I think the primary reason we look at QoS within single administrative domains is because "ALL the operators and providers" is a goal that is quite difficult to model or attain.  My experience with real-time voice  and video services tells me that quite a lot of that traffic originates or is terminated in enterprise or home networks.

<dl> Absolutely agree -  I have many years of experience of trying to deliver dependable real-time voice within Enterprise networks and yes, the only reason it works is because available resources can be matched to customer need and session allowed/denied appropriately.  Much of the configuration is static though and only really works where the traffic is deterministic as is the case for known-codec services.</dl>

If you look at WebRTC-based systems, you'll see something a bit different.  The configurations are not static, but rely on SIP signalling and the selected best-effort path (which can include STUN/TURN servers) being sufficiently well-provisioned that the connection is usable.  Depending on those conditions, you can get quite variable quality (from audio-only to crisp 4k), but it works, even across long paths and at costs that amount to free.  That has been pretty compelling.

<dl> The concern I have is that we are not talking about absolutes and we should be.  If I engineer a system, given the variables, I should know (and be able to predict) exactly the outcome and performance of the system.  We seem to have far too much ‘finger in the air’ stuff going on – this is a branch of engineering and should be as-defined as any other.  Maybe I’m a dreamer… </dl>


If you model a special effort flow as requiring the potential cooperation of every operator of any enterprise or home network, you are going to have a difficult time.  But as soon as you cut them out of the delivery analysis you cut out many real problems in attaining specific metrics (since bufferbloat or other artifacts may be in those last mile networks).   Inside a datacenter, or cable plant, or cellular network you can actually analyze the system well enough to work out whether the costs are worth offering the service. Across the full range of IP networks, I think you will rapidly end up back where we are:  a small set of active markings which are ordinal in character rather than guarantees.

<dl> So this is what I don’t quite get.  If the customer/application declares their requirement and that can be broken down individual service assets that each actor in the infrastructure can understand, then I don’t see why we can’t abstract the notion of service from the underlying technology.   We did this very successfully with SS7 and we do it with IMS (which IS supposed to be multi-domain).

So, I personally think reflecting on SS7 and circuit switched networks doesn't really belong in a discussion of APN, though I'd be happy to share a cup of tea and discuss admission control vs. congestion control at some later time.  But I think the IMS example actually points in an interesting direction, because a lot of systems engineering went into IMS to create distinct services (both voice/real time vs. data and then later slices), but the reality is that many of the use cases that were proposed as requiring those actually end up being met by over-the-top applications that treated the cellular network like a bitpipe.  It's a good enough bitpipe that those work without all of the extra work, and that's a pretty important insight.

<dl> But there is a price-to-pay. If I use IMS within a domain I have an absolute linkage between the access capabilities and the network and we get defined quality.   If I use an OTT service on the default bearer, all bets are off when it comes to quality (or even if it works at all).

The problem is that the OTT solution is essentially dining-out on over-provisioning and this isn’t efficient from a system design point-of-view.  Also, the predominant access method is very limited (and very expensive) spectrum so ideally we need to manage that spectrum to the nth degree…

This is VERY noticeable in cell-edge areas where 4G is still (just) available – VoLTE will almost always be possible because it will run over a defined numerology as controlled by IMS call admission but a WhatsApp voice call will rapidly break down because the numerology if not able to support the data rate and there is no linkage between application and access. </dl>

We do this in many other walks-of-life (railways would be the example that comes to mind as the obvious parallel with multiple services running over co-ordinated infrastructure from different providers but any supply-chain essentially functions as a set of deliverables across multiple actors) so why not the Internet? </dl>


Because dynamic routing across multiple ASNs with half the world's population as a potential end point makes tight coordination more difficult than it is profitable. I'm not personally sure it's possible, but I am very confident that it would cost a mint to deliver if it were, and that the customers wouldn't pay enough to make it worth it.

<dl> It is a complex system but it isn’t an unknown system.  If we had a method for describing required outcomes in terms that could be translated to specific parts of the infrastructure, I don’t see why we can’t have an OTT reservation/brokering system across multiple domains.  The key is ensuring that we have deep enough domain knowledge and control and that IS where APN could be useful.  </dl>

I'm not sure I see the need for the APN work at all, but I would be quite opposed to increasing its scope to include cross-domain coordination, contracts, or settlements.

<dl> I’m not suggesting expanding the scope of APN as a potential WG but posing the question of is there somewhere else we DO need to think about how the system is designed/constructed and whether, end-to-end, it is able to deliver on customer outcomes?

A personal eye-opener for me was when pandemic lockdowns meant that I couldn’t do the two things I love – piano accompanying/teaching and choral singing.  In terms of music, the processing time (musicians refer to it as M2E or Mouth to Ear) is at most 11ms with a variance of well under 1ms.  Really difficult for musicians to cope with is anything that changes the timbre mid-stream (that is the annoying codec changes one hears on radio interviews) or tempo slips due to lost/buffered frames.  Whilst we could fake a choir with clever post-production, the Internet is currently incapable of delivering that service real-time between (e.g.) 32 people sitting at home despite the fact that the raw term that most people understand, bandwidth, vastly exceeds the requirement of even the most chatty codec needed for music.  I spent many-an-hour explaining to people why we just didn’t have a system capable of this today.

I just checked, and Stuart Cheshire's rant about bandwidth vs. latency (http://www.stuartcheshire.org/rants/latency.html<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.stuartcheshire.org%2Frants%2Flatency.html&data=05%7C01%7Cd.lake%40surrey.ac.uk%7C73771509b4154f359d7708dab1ac86f8%7C6b902693107440aa9e21d89446a2ebb5%7C0%7C0%7C638017651838342901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UVp7NHThCYfpicgjD2xhgAjznqrmpQOFXgmIuMTFq%2Bs%3D&reserved=0>) is now 26 years old.  It's been a struggle to get folks to understand that the whole time.

<dl> I remember reading this a LONG time ago and I think we’re at the same point; why do I need a 67Mbit/s DSL circuit to transport G.711 voice – why can’t I just have a perfect ~100kbit/s service between consumer and producer for the duration of the session?  Throwing bandwidth at the problem does not make the problem go away, especially if we have bad/no definition of ‘the problem’ to start with.

I really do think there is merit in revisiting some of the ideas around IMS in terms of what it tried to (and does) achieve and see if we can generalise that to such an extent that we could finely manage flows in order to better utilise finite resources but not in this WG…. Where?</dl>


regards,

Ted



So I think we DO have a problem with making ‘the network’ much more aware of what it is the customer wants to achieve but I agree that this is outside the scope of this WG. </dl>

regards,

Ted Hardie


I think it is naïve to believe that the producer and consumer will exist within the same APN-enabled domain which seems to be the assumption of this statement.  I think what we need is an architecture that enables consumers to describe the outcome they want and then have the various actors in the chain from the producer enable that path for the lifetime of that contract, dynamically rather than the static MPLS-TE broad-brushed approach available today.

And for financial settlement to flow to the actors for delivery against that requirement.

I think that APN MAY be a protocol that would enable that, but unless we have some sense of application-vs-resource balancing based on outcome requirements across multiple domains, then I think we’re starting at the bottom and designing up rather than at the top down!

Personally, I’d like to see a much broader architectural/economic view of delivery in the Internet but I appreciate this is probably beyond the scope of a WG.

David

From: Apn <[e-mail]<mailto:[e-mail]>> on behalf of Sankar Ramamoorthi <[e-mail]<mailto:[e-mail]>>
Date: Tuesday, 18 October 2022 at 04:59
To: Donald Eastlake <[e-mail]<mailto:[e-mail]>>, Andrew Alston - IETF <andrew-ietf=[e-mail]<mailto:[e-mail]>>
Cc: [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
Subject: Re: [Apn] APN Charter revision

Just catching up on the APN charter.

A question.

The last paragraph says


“The approaches used to determine APN information must not rely on the knowledge of specific application and must be removed when packets leave the APN domain “


What is the reason for restricting to determine APN information only based on packet header and not on the knowledge of specific application information? If we did that what value would APN information provide over protocol headers such as VXLAN?

If this has been already discussed and agreed upon, please ignore. From the requirements it looked like the real value of APN information would be to build the metadata based on application knowledge.

Thanks,

Sankar



From: Apn <[e-mail]<mailto:[e-mail]>> on behalf of Donald Eastlake <[e-mail]<mailto:[e-mail]>>
Date: Monday, October 17, 2022 at 6:27 PM
To: Andrew Alston - IETF <andrew-ietf=[e-mail]<mailto:[e-mail]>>
Cc: [e-mail]<mailto:[e-mail]> <[e-mail]<mailto:[e-mail]>>
Subject: [EXTERNAL] Re: [Apn] APN Charter revision
Hi, I have done a pass over this text making some minor grammatical simplifications and improvements and trying to avoid some of the AD comments on the earlier version. (For example, some ADs seem to be bothered  by all caps RFC 2119 words
ZjQcmQRYFpfptBannerStart
This Message Is From an Untrusted Sender
You have not previously corresponded with this sender.
ZjQcmQRYFpfptBannerEnd
Hi,

I have done a pass over this text making some minor grammatical simplifications and improvements and trying to avoid some of the AD comments on the earlier version. (For example, some ADs seem to be bothered  by all caps RFC 2119 words in a Charter so I lower cased them, some ADs commented that the last paragraph was unnecessary, so I have substantially re-worded it...) See attached.

Andrew: I suggest you simply adopt whatever of my changes seem good to you.

Thanks,
Donald
===============================
 Donald E. Eastlake 3rd   [phone] (cell)
 2386 Panoramic Circle, Apopka, FL 32703 USA
 [e-mail]<mailto:[e-mail]>


On Mon, Oct 17, 2022 at 2:01 AM Andrew Alston - IETF <andrew-ietf=[e-mail]<mailto:[e-mail]>> wrote:
Hi All,

The following is the now edited APN charter created using text proposed by Peng Shuping to address blocks on the previous iteration of the charter.  I am aiming to get this submitted first thing tomorrow morning (GMT) pending any comments and edits from this list.  As such, I'm submitting this here for any comments/edits before submission. My thanks to Peng Shuping and the proposed APN chairs for the work they have put in to get this done and to provide the text that forms the basis of the below.

Thanks

Andrew

IETF Routing Area Director

-------- Proposed charter below -----------


The Application-aware Networking (APN) Working Group will develop a framework to enable fine-granularity network service provisioning (traffic operations) within the network domain(s) that supports APN (i.e., APN domain). By APN domain we refer to the operator infrastructure where APN is used from edge to edge (ingress to egress) and where the packet is encapsulated using an outer header incorporating the APN information. An APN domain is defined as a Network Operator controlled limited domain, in which MPLS, VXLAN, SR/SRv6 and other tunnel technologies are adopted to provide network services. APN aims to leverage the ability to apply policies to traffic flows entering into the infrastructure (APN domain). For example, at the headend (ingress) traffic is steered into a given path/policy, at a midpoint node the corresponding performance measurement data is collected, and at a service node a given function is executed.

In modern networks, where things such as deterministic networking and networking slicing are required, there is a requirement for more functionality than QoS can provide.  APN aims to address these requirements and further unleash these technologies.  The goal is to provide network operators with the ability to perform fine-granularity network service provisioning, as opposed to course-grained traffic operations.

Various applications being carried by a modern network, including but not limited to, online gaming, video streaming and enterprise video conferencing have more demanding performance requirements than regular traffic.  These include performance requirements related to low network latency and high bandwidth requirements.  To achieve better Quality of Experience (QoS) for end users, the network needs to be able to provide fine-granularity and application group-level SLA guarantees within the network domains supporting APN.  As part of the framework development, the working group will analyze and define the requirements for such fine-grained provisioning.

To fulfill the above requirements, it is envisaged that APN will need to carry additional APN information in packet headers.  The APN information carried in packets will only be applicable within a limited trust domain.  This information will facilitate fine-granularity service provisioning within the domain.  APN Information within the packets is derived from existing information in the packet headers at the edge of the APN domain, according to rules and policies configured on the edge devices, added to the packets, and then processed within the network.  This information is then removed from the packets leave the domain.
The approaches used to determine APN information MUST NOT rely on the knowledge of specific application and MUST be removed when packets leave the APN domain.  Within any developed framework, both security and privacy MUST be carefully considered.

The APN working group is chartered to define a framework for APN, and following the creation of the framework, the working group MUST be re-chartered before further work, if required, can commence.

Milestones
Date         Deliverable and Milestone
Mar 2023    Working Group Adoption of Problem Statement and Use Cases
Mar 2023    Working Group Adoption of Framework draft
Nov 2023    Problem Statement and Use Cases document submit to IESG
Nov 2023    Framework document submit to IESG





--
Apn mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/apn<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fapn&data=05%7C01%7Cd.lake%40surrey.ac.uk%7C73771509b4154f359d7708dab1ac86f8%7C6b902693107440aa9e21d89446a2ebb5%7C0%7C0%7C638017651838342901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FRTHnxSgyGOYu2W1yo%2Fh295JlWpWW3BDVBLz5RxXgDw%3D&reserved=0>
--
Apn mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/apn<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fapn&data=05%7C01%7Cd.lake%40surrey.ac.uk%7C73771509b4154f359d7708dab1ac86f8%7C6b902693107440aa9e21d89446a2ebb5%7C0%7C0%7C638017651838342901%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FRTHnxSgyGOYu2W1yo%2Fh295JlWpWW3BDVBLz5RxXgDw%3D&reserved=0>