Re: [Idnet] IDN dedicated session call for case

Dear Albert, all,

Please find some comments below.

Yassine
_______________________________________
Yassine HADJADJ AOUL

Associate Professor, University of Rennes1
Dionysos Team / IRISA Lab
Room: F334
Phone: +33 (0) 2 99 84 71 35

> Le 9 août 2017 à 04:44, Albert Cabellos <albert.cabellos@gmail.com> a écrit :
> 
> Hi Yanshen
> 
> In my view they are not the same use-case
> 
> - QoS use-case: As far as I understand here you aim to configure the routing/switching infrastructure to achieve QoS for flows.
> 
> - QoE use-case: Here the idea is that QoE can not typically be measured directly over the network, it has to be measured either over the application (e.g., buffering in on-demand video services) or by polling the human users. Thus, measuring QoE is expensive.

That is true that to have a very good accuracy it is generally necessary to have application-level measures (like the quantization parameter in a video, or the playout interruptions). However, there is many papers in the literature, some from our team, which demonstrate a good accuracy for voice and video (UDP or RTP) using only network level parameters.

We considered using Random Neural Networks in our developed tools … and we have ongoing work on that ...

Concerning the cost, it is indeed expensive generally. However, there are some techniques in the literature, which consist in doing the learning with objective techniques (like VQM) … this eliminate the need of users' panel … 

A former PhD student of our team had some contributions on that:
No-reference Quality of Experience estimation of H264/SVC stream
http://ieeexplore.ieee.org/document/6477778/ <http://ieeexplore.ieee.org/document/6477778/>

> 
> What can be measured are the low-level network metrics (delay, jitter, SNR, etc). Then the question is, which is the relation between low-level metrics and QoE metrics? The idea is to create a data-set containing low-level and QoE metrics. Then and thanks to ML we model the relation between low-level and QoE metrics, this means that we understand that when delay<N, jitter<M and SNR>K QoE levels are satisfactory. With this, operators know what low-level performance they need to target to offer good QoE. 

The relation between QoS and QoE is generally non linear, so I believe that in spite of finding a precise target we will have a function, which may predict the QoE … and thus, we may decline some actions accordingly.

Yassine

> 
> A nice relation is that once you establish the target performance of low-level metrics to achieve QoE, you can then use the 'QoS use-case' (or similar) to operate the network.
> 
> Albert 
> 
> On Tue, Aug 8, 2017 at 7:02 PM, yanshen <yanshen@huawei.com <mailto:yanshen@huawei.com>> wrote:
> Dear Albert,
> 
>  
> 
> At least two supporters you have : )
> 
>  
> 
> I think that the QoS and QoE is just similar with my opinion mentioned before that is the data can be divided into subjective and objective.  This will be related with the data format and the way of obtaining. And your case build up a bridge between the subjective and objective.
> 
>  
> 
> Yansen
> 
>  
> 
>  
> 
> From: Özgü Alay [mailto:ozgu@simula.no <mailto:ozgu@simula.no>] 
> Sent: Tuesday, August 08, 2017 2:02 PM
> To: Albert Cabellos <albert.cabellos@gmail.com <mailto:albert.cabellos@gmail.com>>
> Cc: yanshen <yanshen@huawei.com <mailto:yanshen@huawei.com>>; idnet@ietf.org <mailto:idnet@ietf.org>
> Subject: Re: [Idnet] IDN dedicated session call for case
> 
>  
> 
> Dear Albert, 
> 
> We are interested in this use case and will support the activities in this area.
> 
> Best Regards,
> 
> Özgü
> 
>  
> 
> On 8 August 2017 at 06:52, Albert Cabellos <albert.cabellos@gmail.com <mailto:albert.cabellos@gmail.com>> wrote:
> 
> Hi all
> 
>  
> 
> Here´s another use-case:
> 
>  
> 
> Use case N+2: QoE
>         Description: Collect low-level metrics (SNR, latency, jitter, losses, etc) and measure QoE. Then use ML to understand what is the relation between satisfactory QoE and the low-level metrics. As an example learn that when delay>N then QoE is degraded, but when M<delay<N then QoE is satisfactory for the customers (please note that QoE cannot be measured directly over your network). This is useful to understand how the network must be operated to provide satisfactory QoE.
>         Process: 1. Low-level data collection and QoE measurement ; 2. Training Model (input low-level metrics, output QoE); 3. Real-time data capture and input; 4. Predict QoE; 5. Operate network to meet target QoE requirement, go to 3.
>         Data Format:    Time : [Start, End, Unit, Number of Value, Sampling Period]
>                                 Position: [Device ID, Port ID]
>                                 Direction: IN / OUT
>                                 Low-level metric : SNR, Delay, Jitter, queue-size, etc
> 
> 
>         Message :       Request: ask for the data
>                                 Reply: Data
>                                 Notice: For notification or others
>                                 Policy: Control policy
> 
>  
> 
> Kind regards
> 
>  
> 
> Albert
> 
>  
> 
> On Wed, Aug 2, 2017 at 7:12 PM, yanshen <yanshen@huawei.com <mailto:yanshen@huawei.com>> wrote:
> 
> Dear all,
> 
> Since we plan to organize a dedicated session in NMRG, IETF100, for applying AI into network management (NM), I’d try to list some Use Cases and propose a roadmap and ToC before Nov.
> 
> These might be rough. You are welcome to refine them and propose your focused use cases or ideas.
> 
> Use case 1: Traffic Prediction
>         Description: Collect the history traffic data and external data which may influence the traffic. Predict the traffic in short/long/specific term. Avoid the congestion or risk in previously.
>         Process: 1. Data collection (e.g. traffic sample of physical/logical port ); 2. Training Model; 3. Real-time data capture and input; 4. Predication output; 5. Fix error and go back to 3.
>         Data Format:    Time : [Start, End, Unit, Number of Value, Sampling Period]
>                                 Position: [Device ID, Port ID]
>                                 Direction: IN / OUT
>                                 Route : [R1, R2, ..., RN]  (might be useful for some scenarios)
>                                 Service : [Service ID, Priority, ...]  (Not clear how to use it but seems useful)
>                                 Traffic: [T0, T1, T2, ..., TN]
>         Message :       Request: ask for the data
>                                 Reply: Data
>                                 Notice: For notification or others
>                                 Policy: Control policy
> 
> Use case 2: QoS Management
>         Description: Use multiple paths to distribute the traffic flows. Adjust the percentages. Avoid congestion and ensure QoS.
>         Process: 1. Data capture (e.g. traffic sample of physical/logical port ); 2. Training Model; 3. Real-time data capture and input; 4. Output percentages; 5. Fix error and go back to 3.
>         Data Format:    Time : [Timestamp, Value type (Delay/Packet Loss/...), Unit, Number of Value, Sampling Period]
>                                 Position: [Link ID, Device ID]
>                                 Value: [V0, V1, V2, ..., VN]
>         Message :       Request: ask for the data
>                                 Reply: Data
>                                 Notice: For notification or others
>                                 Policy: Control policy
> 
> Use case N: Waiting for your Ideas
> 
> Also I suggest a roadmap before Nov if possible.
> 
> ### Roadmap ###
> Aug. : Collecting the use cases (related with NM). Rough thoughts and requirements
> Sep. : Refining the cases and abstract the common elements
> Oct. : Deeply analysis. Especially on Data Format, control flow, or other key points
> Nov.: F2F discussions on IETF100
> ### Roadmap End ###
> 
> A rough ToC is listed in following. We may take it as a scope before Nov. Hope that the content could become the draft of draft.
> 
> ###Table of Content###
> 1. Gap and Requirement Analysis
>         1.1 Network Management requirement
>         1.2 TBD
> 2. Use Cases
>         2.1 Traffic Prediction
>         2.2 QoS Management
>         3.3 TBD
> 3. Data Focus
>         3.1 Data attribute
>         3.2 Data format
>         3.3 TBD
> 4. Aims
>         4.1 Benchmarking Framework
>         4.2 TBD
> ###ToC End###
> 
> 
> Yansen
> 
> _______________________________________________
> IDNET mailing list
> IDNET@ietf.org <mailto:IDNET@ietf.org>
> https://www.ietf.org/mailman/listinfo/idnet <https://www.ietf.org/mailman/listinfo/idnet>
>  
> 
> 
> _______________________________________________
> IDNET mailing list
> IDNET@ietf.org <mailto:IDNET@ietf.org>
> https://www.ietf.org/mailman/listinfo/idnet <https://www.ietf.org/mailman/listinfo/idnet>
>  
> 
> 
> _______________________________________________
> IDNET mailing list
> IDNET@ietf.org
> https://www.ietf.org/mailman/listinfo/idnet