Re: [alto] qoRE: Pub/sub thread; Was Re: ALTO Draft ReCharter WG review(Internet mail)

["chunshxiong(熊春山)" <[e-mail]>]

Hello Richard,

Yes, the two use cases can be use the same “sub/pub” mechanism to subscribe the requirement to the ALTO Server and the ALTO Server can notify the (quick) change of the bitrate/latency of network to the ALTO Client.

I just use “sub/pub” to demonstrate the procedures to “request “ and “notify” , how to support these procedures with current protocol if the current protocol can support these procedure  or whether new protocol should be defined to support need to be identified ?

We do not need re-invent the wheel, but if the current tools can not support the use cases, we need to define new protocol.


BRs,
Chunshan

From: Y. Richard Yang <[e-mail]>
Sent: Friday, March 12, 2021 3:55 AM
To: chunshxiong(熊春山) <[e-mail]>
Cc: Li Gang <[e-mail]>; [e-mail]; [e-mail]; Kai Gao <[e-mail]>; Qin Wu <[e-mail]>; IETF ALTO <[e-mail]>
Subject: Re: qoRE: [alto] Pub/sub thread; Was Re: ALTO Draft ReCharter WG review(Internet mail)

Hi Chunshan,

Thanks a lot for the clarification. Please see below.

On Wed, Mar 10, 2021 at 2:31 AM chunshxiong(熊春山) <[e-mail]<mailto:[e-mail]>> wrote:
Hello all,

There are a lot of interesting discussion on the “pub/sub” related use cases and mechanisms.

Here I want to provide some use cases on how the “pub/sub” needs to support:


1)     One sub/pub for different bitrates :

e.g.  the alto client can subscribe the alto server to get 5G network current supporting bitrates as :

a) bigger than  A kbps;

b) less than A but bigger than B;

c) less than B;

where A>B, e.g. A is for 1080p video streaming, B is for 720p streaming,

of course, we can introduce more different bitrates span, but for simple, only 3 are defined in the example.



In such case the alto server will pub(notify) the alto client the current 5G network bitrates when the bitrates span is changed (not the bitrates are changed, only after the bitrate changed into different bitrate span).

It is each to above bitrates to latency.
This is a good use case. It is essentially a dampening mechanism to reduce the load on the (server-client) push channel.

Another related comment is that I may need to finalize the term for this type of communication. It may not be pub/sub, but something different. Traditional ALTO request-response (without SSE) is a "simple" request-response, in which the client sends a request, the server receives the request, generates a response, and sends the response back to the client; the connection is conceptually closed. Your setting is an example that is getting closer to a more general "RPC" model, for example, the protocol buffer model (https://developers.google.com/protocol-buffers/docs/reference/proto3-spec):

service = "service" serviceName "{" { option | rpc | emptyStatement } "}"
rpc = "rpc" rpcName "(" [ "stream" ] messageType ")"
              "returns" "(" [ "stream" ] messageType ")"
   (( "{" {option | emptyStatement } "}" ) | ";")

It looks that you are asking to add the "stream" in return.


2)     Normally, the smart UE-client and cloud Server application can use the adaptive bitrate changes to handle the network bitrate changes as above, but there are still a lot of chances that the client application temporary halts to wait the downlink data in the cloud gaming. Because in some case the bitrates of the 5G RAN change very quicky and steeply in a short time and the ALTO server can not in time to provide such information to the alto client.

In order to handle such steeply and quickly bitrate changes in the wireless system, a new sub/pub use case is needed as below.

The alto client can subscribe the ALTO server to notify the Quick QoS Change. E.g. if the bitrate decreases/increases  30% (e.g. changes +- 30%) （ or decreases/increases  A kbps） in a short time ，the alto serve shall notify the alto client very quickly and emergently.

(Normally, if the bitrates changes  up and down very quickly but the average bitrate are not changed, it is not considered as the QUICK QOS Change. The QUICK QoS Change means the bitrates are changed quickly and steeply in short time and keeps the new bitrates  after the QUICK change).

With such QUICK QoS Change sub/pub, the cloud gaming server can real-time to get the 5G network bitrates big changes and quickly use the adaptive encoding scheme.
What is the differences between the above 1) and 2) ?
Normally the 1) needs the alto client to provides the bitrates spans to the alto server and 5G network , and 5G network or the alto server  needs to compare its current bitrates with the provided bitrate span. But for the 2) the alto client does not provide any defined bitrate values, it only provides the bitrate change ratio or change values to the alto server and 5G, then the 5G network or the alto server needs to compare the bitrate changes and notify the alto client.

1)     is well defined in 3GPP for 5G standards, but this method is used only for the Guaranteed QoS Flow (currently, the Guaranteed  QoS Flow is used by is not used very widely, currently it is only used by the operator’s HD Voice (i.e. Voice Over LTE/5G, i.e. Voice over IMS) service ); 2) can be used by the Guaranteed QoS Flow and non-Guaranteed QoS Flow (the Non Guaranteed QoS Flow is widely used by the internet applications ). We/Tencent are going to study the 2) and try to find out how much QoE improvement based on this Quick QoS Change. We/Tencent also are planning to push 3GPP 5G to study this Quick QoS Change standards in Release 18.

Thank you for clarifying these two different use cases.

My first reaction is that regardless of which use case, they are good use cases supporting the ALTO mission: network providing information that the clients may not easily get by themselves. The wireless channels are complex, expensive, shared channels. It is a location where the network (ALTO) server can provide highly valuable information, but the client may not obtain it easily (due to more limited state access, limited computation power, the latency involved, ...)

Now back to your two use cases, I see that they can be supported using a single mechanism (i.e., language grammar). A network may provide one or the other, and the WG defines the grammar. Is this acceptable?

Richard

BRs,
Chunshan Xiong

From: alto <[e-mail]<mailto:[e-mail]>> On Behalf Of Y. Richard Yang
Sent: Wednesday, March 10, 2021 12:13 PM
To: Li Gang <[e-mail]<mailto:[e-mail]>>
Cc: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; Kai Gao <[e-mail]<mailto:[e-mail]>>; Qin Wu <[e-mail]<mailto:[e-mail]>>; IETF ALTO <[e-mail]<mailto:[e-mail]>>
Subject: [alto] Pub/sub thread; Was Re: ALTO Draft ReCharter WG review(Internet mail)

Dear all,

I renamed the generic email Subject line so that we may focus this thread on item 2.

Here are two quick comments related with this thread, and I will go to more details as soon as I can.

1. ALTO SSE appears to be a quite useful service and it helps to adapt it to use more modern transport such as HTTP/2, quic.

We may consider this under the sub/pub umbrella by considering the service as a special sub/pub:
ALTO client can manage its subscription of network info to be pushed to it, where the granularity of subscription is individual ALTO resource query; in this model, the granularity can be quite fine-grained, depending on the query granularity. Hence, if we want fine-grained subscriptions, we can introduce these queries and leave the push/incremental encoding mechanism/framework alone.

2. Can we clarify more on the sub/pub use cases? A typical use case of a sub/pub system C is to facilitate the communication of other entities such as A and B. Hence, if the ALTO server is C, then who are the A and B? I can think of use cases where clients, for example, in a mobility/weakly connected setting, are A and B. Or the use case is about developing C, and the A and B are ALTO client/server. It helps to clarify.

Richard

On Sun, Mar 7, 2021 at 10:42 PM Li Gang <[e-mail]<mailto:[e-mail]>> wrote:
Hi, Qin,
Please see my reply inline.

Li Gang

From: Qin Wu [mailto:[e-mail]<mailto:[e-mail]>]
Sent: Monday, March 8, 2021 10:52 AM
To: Li Gang; [e-mail]<mailto:[e-mail]>
Cc: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; 'IETF ALTO'
Subject: RE: [alto] ALTO Draft ReCharter WG review

Hi, Gang:
Thanks for sharing your use case, let me rephrase what you envision for your use case,
You want to express QoS requirement in the subscription request, the network exposes the network information via notification in response to subscription request,
application operators can tune adaptive rate to improve user QoE based on the network information change.

[Gang]: yes

Can you clarify a little bit about specific application traffic patterns?

[Gang]: let me take video streaming as an example, normally the downlink streaming content would be segmented into pieces for `10 seconds. For each piece, multiple video encoding rates, for example 1080p, 720p …, can be provided and adjusted by server. For each encoding rate, the QoS requirement (e.g. throughput, latency) is different. The network can provide such information change  (e.g. whether QoS requirement for 1080p, 720p is fulfilled or not) via pub/sub, which help application operator tune encoding rate.

Secondly, I agree fine granularity pub sub can consider one time subscription and configure wait time as subscription policy to alleviate the signaling load on the network.

-Qin
发件人: Li Gang [mailto:[e-mail]<mailto:[e-mail]>]
发送时间: 2021年3月7日 16:30
收件人: [e-mail]<mailto:[e-mail]>; Qin Wu <[e-mail]<mailto:[e-mail]>>
抄送: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; 'IETF ALTO' <[e-mail]<mailto:[e-mail]>>
主题: RE: [alto] ALTO Draft ReCharter WG review

Hi, Kai and Qin,

Thanks for triggering the discussion on  the 2nd item of the recharter text.
I agree that it would be better to define a generic pub/sub framework irrespective of specific transport protocol.
We can start with a simple pub/sub mechanism, which is driven by concrete use cases and then consider to extend as needed.

Some of my thoughts are inline.

Li Gang

From: alto [mailto:[e-mail]] On Behalf Of [e-mail]<mailto:[e-mail]>
Sent: Friday, March 5, 2021 11:03 AM
To: Qin Wu
Cc: [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>; IETF ALTO
Subject: Re: [alto] ALTO Draft ReCharter WG review


Hi Qin,

Thanks for the comments. A quick summary of my response is

1. "Pub/sub" means different things in different contexts and I think we must clarify what it means in the context of distributing ALTO information.

2. There are two ways of realizing complex "pub/sub" of ALTO information but I think they are fundamentally different deployment settings for one generic framework (whose details are, unfortunately, not thought through yet).

Please see the details inline.

Best,

Kai
-----Original Messages-----
From:"Qin Wu" <[e-mail]<mailto:[e-mail]>>
Sent Time:2021-03-04 22:21:06 (Thursday)
To: "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>
Cc: "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>, "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>, "IETF ALTO" <[e-mail]<mailto:[e-mail]>>
Subject: Re: [alto] ALTO Draft ReCharter WG review
Kai:
发件人: [e-mail]<mailto:[e-mail]> [mailto:[e-mail]]
发送时间: 2021年3月3日 21:40
收件人: Qin Wu <[e-mail]<mailto:[e-mail]>>
抄送: IETF ALTO <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>; [e-mail]<mailto:[e-mail]>
主题: Re: [alto] ALTO Draft ReCharter WG review


Dear all,

 Below are some comments on the 2nd item in the recharter text.

As far as I know, the ALTO incremental update extension (RFC 8895) already provides a mechanism to enable the "pub-sub" of ALTO information, using Server-Sent Events (SSE). I see there are multiple directions indicated by the new charter item:
[Qin]: Thanks for clarifying the difference between SSE and pub sub proposed in the new proposed charter item.

1. Decouple the "pub-sub" protocol with the underlying mechanism.

Besides SSE, other mechanisms can also be used to realize the "pub-sub" of ALTO information, such as HTTP/2, HTTP/3 or the methods mentioned in the charter text. Thus, a direct extension is to define the abstract format of control messages and data messages (i.e., WHAT information should be provided but not HOW), and allow different underlying protocols to use protocol-specific encodings.

For example, SSE encodes the metadata (e.g., content-type and stream id) and the content of an event using "event:" and "data:" prefixes at the beginning of each line, and uses empty lines to indicate the end of a message, while HTTP/2 (RFC 7540) may encode the metadata and the content of an event using PUSH_PROMISE/HEADERS and DATA frame .

[Qin]: Good analysis, I think we need to decide whether we should define generic pub sub mechanism or transport specific pub sub mechanism. Do you have any suggestion on this?

[KAI]: I think the generic pub-sub mechanism (or maybe the term framework is more appropriate) is more important at this point, which should also cover the direction of providing more fine-grained control. One thing that just strikes me after taking a quick look at rabbitMQ is that "pub/sub" means different things in different contexts. It is important we understand what are the requirements of generic pub/sub in the ALTO framework.

[KAI]: When we discuss "pub/sub" with SSE and HTTP/2, which is a one-to-one client-server communication, the focus of the "pub/sub" here is simple: what are the messages and how the client can control the subscribed information. However, with message queues (e.g., rabbitMQ), the communication pattern may be more complex: a message can be sent to multiple queues without knowing exactly who is subscribing. I see two ways to realize the more complex "pub/sub" requirement for ALTO information.

rabbitMQ: https://www.rabbitmq.com/tutorials/tutorial-three-python.html

[KAI]: CASE A: First, the client application may deploy its own "pub/sub" system, and the ALTO client simply serves as a producer by forwarding the ALTO messages to the "pub/sub" system. In this way, the problem is reduced to the one-to-one "pub/sub" problem.

[KAI]: CASE B: Second, ALTO servers may natively support the "pub/sub" of ALTO information. In this case, an ALTO server may need to handle events such as join/leave of clients that subscribe to the same ALTO information. For example, for a client that just subscribes to a network map, the server should send the whole map instead of incremental updates.

[KAI]: Both approaches have pros and cons. The first is simple on the server side but may be less efficient (because of triangle routing) and complex on the client side (client must handle data consistency to support dynamic subscribers). I think the generic framework should contain two aspects:

[KAI] 1. Control of ALTO information: a server-client protocol which is similar to RFC 8895 but maybe with some extended capabilities.

[KAI] 2. Distribution of ALTO information: a MQ-like protocol that controls how the ALTO information can be efficiently and consistently delivered to subscribers.

[KAI] I think the connection between these two aspects is a logical entity called ALTO Exchange (following the term used by rabbitMQ). This entity can be operated by an application provider (as in CASE A) or by a network operator (as in CASE B). However, the detailed responsibilities of this entity may still need some investigation.

I think this requirement may help integrating ALTO in network management platforms (such as OpenDaylight, Kubernetes, and ETSI ZSM*) which design their own pub-sub systems for reasons such as consistency or ease of development. It would be great if there is an interest in this direction from companies/organizations.

[Qin]: I can see The 3GPP has defined a Service-Based Architecture (SBA), whereby the control plane functionality and common data repositories of a 5G network are delivered by way of a set of interconnected Network Functions (NFs),pub sub mechanism has been well adopted in 3GPP interface.

Also in the public cloud, popular pub/sub implementations has been widely deployed,e.g., rabbitMQ (AMQP), mosquitto (MQTT), ejabberd (XMPP), and ZeroMQ. We also see many pub sub mechanism or extension has been developed in IETF, e.g., YANG Push, draft-ietf-dots-telemetry, draft-ietf-ace-mqtt-tls-profile.

* The integration fabric of ETSI ZSM provides pub-sub support but ZSM also allows services to use their own pub-sub mechanisms.

2. Enable more fine-grained control of pub-sub.

In RFC 8895, there are two types of commands which only defines WHAT information to subscribe:

- add: Make one or more new requests to receive the incremental updates.

- remove: Terminate the subscription of one or more previously-made requests.

In the meantime, the updates will be continuously sent to the client whenever a server sees fit.

The charter text proposes to enable ALTO clients to request and receive "a diverse types (such as event-triggered/sporadic, continuous), continuous, customized feed of publisher-generated information". It seems to me that the new extension wants to allow clients to specify not only WHAT information to be subscribed but also WHEN/HOW the information should be delivered (e.g., Notify me the latest value every 5 second.).

[Qin]:Good point, I think fine grained control of pub-sub allows not only periodical subscription, but also on demand subscription, which is the missing piece in the existing SSE incremental update.

[Gang]: I think WHAT/WHEN/HOW should all be considered for pub/sub. Specifically, after one time subscription, notification may be one time, multiple times based on either event triggering or periodic reporting. Besides that, in order to reduce signaling overhead, maybe minimum wait time can be defined between consecutive pub messages.

Personally I find both directions to be interesting and useful. It would be great if they can be supported by real use cases.

[Gang]: I also agree that the necessary extension should be well supported by concrete use cases. I’d like share some thoughts on use cases in my mind: some application operators would like to subscribe the QoS requirements（maybe one or a list）for their specific application traffic patterns to the network operators, the network can provide notification whether/how the network would fulfill their requirements, so that application operators can perform adaptive rate control to improve user QoE.

Just my two cents.

Best,

Kai
-----Original Messages-----
From:"Qin Wu" <[e-mail]<mailto:[e-mail]>>
Sent Time:2021-02-22 21:50:44 (Monday)
To: "IETF ALTO" <[e-mail]<mailto:[e-mail]>>
Cc: "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>, "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>
Subject: [alto] ALTO Draft ReCharter WG review
Hi, :
We have requested one hour session for ALTO WG meeting in the upcoming IETF 110, which is arranged on Friday, March 12, 14:30-15:30(UTC).
The goal is to boil down ALTO recharter and have consensus on charter contents in IETF 110.
To get this goal, an updated inline draft charter text for ALTO has just been posted to this list,

This charter has received a couple of rounds of informal review from WG members, chairs and our Ads from brief to deep thorough, 5 new chartered items have been listed.
We would like to solicit feedback on these new chartered items and your use case, deployment, idea corresponding to these new chartered items.
Sharing your past deployment story will also be appreciated.

============================================================================================
The ALTO working group was established in 2008 to devise a request/response protocol to allow a host to benefit from a server that is more cognizant of the network infrastructure than the host is.

The working group has developed an HTTP-based protocol and recent work has reported large-scale deployment of ALTO based solutions supporting applications such as content distribution networks (CDN).

ALTO is now proposed as a component for cloud-based interactive applications, large-scale data analytics, multi-cloud SD-WAN deployment, and distributed
computing. In all these cases, exposing network information such as abstract topologies and network function deployment location helps applications.

To support these emerging uses, extensions are needed, and additional functional and architectural features need to be considered as follows:

o Protocol extensions to support a richer and extensible set of policy attributes in ALTO information update request and response. Such policy attributes may indicate information dependency (e.g., ALTO path-cost/QoS properties with dependency on real-time network  indications), optimization criteria (e.g., lowest latency/throughput network performance objective), and constraints (e.g., relaxation bound of optimization criteria, domain or network node to be traversed, diversity and redundancy of paths).

o Protocol extensions for facilitating operational automation tasks and improving transport efficiency. In particular, extensions to provide "pub/sub" mechanisms to allow the client to request and receive a diverse types (such as event-triggered/sporadic, continuous), continuous, customized feed of publisher-generated information. Efforts developed in other working groups such as MQTT Publish / Subscribe Architecture, WebSub, Subscription to YANG Notifications will be considered, and issues such as scalability (e.g., using unicast or broadcast/multicast, and periodicity of object updates) should be considered.

o The working group will investigate the configuration, management, and operation of ALTO systems and may develop suitable data models.

o Extensions to ALTO services to support multi-domain settings. ALTO is currently specified for a single ALTO server in a single administrative domain, but a network may consist of
multiple domains and the potential information sources may not be limited to a certain domain. The working group will investigate extending the ALTO framework to (1) specify multi-ALTO-server protocol flow and usage guidelines when an ALTO service involves network paths spanning multiple domains with multiple ALTO servers, and (2) extend or introduce ALTO
services allowing east-west interfaces for multiple ALTO server integration and collaboration. The specifications and extensions should use existing services whenever possible. The specifications and extensions should consider realistic complexities including incremental deployment, dynamicity, and security issues such as access control, authorization (e.g., an ALTO server provides information for a network that the server has no authorization), and privacy protection in multi-domain settings.

o The working group will update RFC 7971 to provide operational considerations for recent protocol extensions (e.g., cost calendar, unified properties, and path vector) and new extensions that the WG develops. New considerations will include decisions about the set of information resources (e.g., what metrics to use), notification of changes either in proactive or reactive mode (e.g., pull the backend, or trigger just-in-time measurements), aggregation/processing of the collected information  (e.g., compute information and network information )according to the clients’ requests, and integration with new transport mechanisms (e.g., HTTP/2 and HTTP/3).

When the WG considers standardizing information that the ALTO server could provide, the following criteria are important
to ensure real feasibility:

- Can the ALTO server realistically provide (measure or derive) that information?

- Is it information that the ALTO client cannot find easily some other way?

- Is the distribution of the information allowed by the operator of the network? Does the exposure of the information introduce privacy and information leakage concerns?

Issues related to the specific content exchanged in systems that make use of ALTO are excluded from the WG's scope, as is the issue of dealing with enforcing the legality of the content. The WG will also not propose standards on how congestion is signaled, remediated, or avoided.

-Qin Wu (on behalf of chairs)
_______________________________________________
alto mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/alto


--
--
 =====================================
| Y. Richard Yang <[e-mail]<mailto:[e-mail]>>   |
| Professor of Computer Science       |
| http://www.cs.yale.edu/~yry/        |
 =====================================