Re: [xrblock] Call for adopting xr-quality-monitoring-06 as WG item

[Colin Perkins <[e-mail]>]

Qin,

See inline.
Colin


On 20 Dec 2011, at 01:54, Qin Wu wrote:
> Hi, Colin:
> I understand your concerns. It is painful, but that's the tradeoff we should made.
> Please see my clarification belows.
> ----- Original Message ----- 
> From: "Colin Perkins" <[e-mail]>
> To: "Shida Schubert" <[e-mail]>
> Cc: <[e-mail]>
> Sent: Tuesday, December 20, 2011 1:13 AM
> Subject: Re: [xrblock] Call for adopting xr-quality-monitoring-06 as WG item
> 
> 
>> Shida,
>> 
>> I don't think this draft is ready to adopt:
>> 
>> - The single-stream metric mixes audio and video MOS into a single metric block, with a grab-bag of MOS type and calculation algorithm parameters. It seems that little thought has gone into designing the XR block format to prohibit invalid combinations (e.g., MT = MOS-V with CALg = G.107), and there is no guidance on which algorithm is appropriate for which MOS type. 
> 
> [Qin]:  We do have guidance to prevent such combination  in three places
> a. See MoS Type (MT) field definition, it said:
> "
>      If MoS type is MoS-LQ and MoS-CQ, the MoS value can be calculated
>      based on ITU-T G.107[G.107], ITU-T P.564 [P.564]or ETSI TS 101
>      329-5 [ETSI], if the Mos type is MoS-V or MoS-AV, the Mos value
>      can be calculated based on ITU-T P.NAMS [P.NAMS]or ITU-T P.NBAMS
>      [P.NBAMS]. 
> "
> b. Also see Mos Type (MT) field defintion, it said:
> "
>     If MoS Type does not match the MoS algorithm in the report (e.g.,
>      specify a voice MOS algorithm for a video quality MOS), the
>      receiver should also discard this report.
> "
> c.  See Calculation Algorithm (CALg) field definition
> It distinguish Audio MoS algorithm from video MoS algorithm by adding "(voice)" into Audio MoS algorithm.

I agree that there is guidance to avoid invalid combinations. However, a better design would prevent invalid combinations by appropriate choice of report block structure. 

I still see no guidance on which algorithm is appropriate for which MOS type.

>> The format seems to follow the RFC 3611 model, throwing together a set of unrelated parameters, rather than the approach documents in the monarch draft.
> 
> [Qin]: In this draft, we are not allowed to put "single stream segment","multi-layer segment" and "multi-channel segment" into one XR Block Report. please see the note in the segement i field definition
> it said:
> "
>      Note that in these
>      three segment type,any two segment types can not be present in the
>      same metric block.
> "
> That is to say if we choose to use "single stream segment" to report MoS, we will not use "multi-channel segment" or "multi-layer segment" . If we choose to use multi-channel segment, we will not use 
> "single stream segment" or "multi-layer segment".
> 
> Also we may need to measure MoS-CQ using three different audio MoS algorithms, e.g., 
> Using ITU-T P.564 to calculate MoS-CQ1
> Using G.107 to calculate MoS-CQ2
> Using ETSI TS 101 329-5 to calculate MoS-CQ3
> 
> In this case,  MoS-CQ1 and MoS-CQ2 and MoS-CQ3 are all MoS value, but using different audio MoS algorithm, so the question is how to report them, we have two options:
> 1. Report each MoS-CQ value carrried in the "single stream segment" using three XR Blocks 
> 2. Report three MoS-CQ vlaues carried in three "single stream segment" using only one XR Block.
> I think the option 2 is a better approach, this approach only report MoS value related to one single MoS Type and doesn't put a set of unrelated parameters together and therefore doesn't conflict the rule we set in MONARCH.
> However if we also measure MoS-LQ using three different audio MoS algorithms and report MoS-CQ value and MoS-LQ value carried in two "single stream segment" using one single XR Block, that does mean "put a set of unrelated parameters together" and therefore conflict with Monarch.
> 
> So my suggestion is to add a sentence to say only one MoS type is allowed to report in one single XR Block.
> 
> If you agree with this, I can revise the draft to make this clear. 

This change doesn't address my concern, which is that a single draft is describing three different types of report block. The monarch draft was intended to recommend different report blocks be developed in different drafts, so they can be implemented separately, in a mix-and-match style, rather than being combined into a single draft.

>> - The multi-layer segment is ill-specified: does it support audio only, video only, or both depending on the Media Type bit? 
> 
> [Qin]: multi-layer segment does support both video and audio, 

None of the field descriptions in Section 3.2.2 mention audio; they're all video only.

>> Which calculation algorithm makes sense here, since there is no guidance given, and it's possible to specify any algorithm? 
> 
> [Qin]: currently we only allow 5 MoS algorithms.since three Audio MoS algorithms have already been standardized and the other two are in standardization progress.

This doest address my question. What algorithms make sense in the context of layered media?

>> What is the SSID field? How it is used? 
> 
> [Qin]:SSID is sub stream identifier and can be used to identifier each layer if all the layers are carried in the same RTP stream.

More detail is needed in the draft.

>> It's not sufficient to simply say that a "NAL unit type is one example", the draft needs to give normative rules for the use of this field.
> 
> [Qin]:Agree.
> 
>> - The media type, MOS type, and calculation algorithm for the multi-channel segment are ill-specified in similar ways to the multi-layer segment.
> 
> [Qin]: Yes, you are right.
> 
>> Also, it's not clear that the channel mapping in RFC 3550 Section 4.1 is the only one in use.
> 
> [Qin]: Good point, but that's what we have in mind currently.
> 
>> - In general, it's not clear to me that including multiple different types of MOS report (single-stream, multi-layer, and multi-channel) in a single draft fits with the monitoring architecture, which encourages small, single-use, XR block definitions. I would suggest that the three different MOS report types would be better defined as three drafts: there is little in common between them.
> 
> [Qin]: As I clarified above, these three MOS report types can not be put into one single XR Block.They are excluded between one another. 
> But If WG think it is a good idea to merge them together, I am okay with that.

I agree that they should not be included in a single XR block. My question, however, was whether they should be in a single draft. 

>> - The draft provides no meaningful guidelines for registration of new MOS algorithms, and has an inadequate IANA registry.
> 
> [Qin]: We do have in the (-06), section 5.4 of  xr-quality-monitoring-06

Much more detail on the registration procedures, and guidelines for what should be registered, is needed.


> "
> 5.4.  New registry of calculation algorithms
> 
>   This document creates a new registry to be called "RTCP XR QoE metric
>   block - Calculation Algorithm" as a sub-registry of the "RTP Control
>   Protocol Extended Reports (RTCP XR) Block Type Registry".  Policies
>   for this new registry are as follows:
> 
>   o  Entries in the registry are integers.  The valid range is 0 to 7
>      corresponding to the 3-bit field "CAlg" in the block.  Values are
>      to be recorded in decimal.
> 
>   o  Initial assignments are as follows:
> 
>      1.  ITU-T P.564 Compliant Algorithm [P.564] (Voice)
>      2.  G.107 [G.107] (Voice)
>      3.  ETSI TS 101 329-5 Annex E ETSI] (Voice)
>      4.  ITU-T P.NAMS [P.NAMS]
>      5.  ITU-T P.NBAMS [P.NBAMS]
> "
> Hope this makes sense.
> 
>> Colin
>> 
>> 
>> On 19 Dec 2011, at 11:55, Shida Schubert wrote:
>>> This is a call to see whether people are happy 
>>> with the latest xr-quality-monitoring-06 draft for addressing 
>>> the milestone "QoE Metrics Reporting".
>>> 
>>> http://tools.ietf.org/html/draft-wu-xrblock-rtcp-xr-quality-monitoring-06
>>> 
>>> xr-que-00 which used to be an AVT draft also addresses 
>>> the same milestone. 
>>> 
>>> http://tools.ietf.org/id/draft-clark-xrblock-rtcp-xr-qoe-00.txt
>>> 
>>> The xr-quality-monitoring-06 claims to merge the two 
>>> into one. 
>>> 
>>> Please response to express your option for the followings. 
>>> 
>>> 1. Adopt the draft-wu-xrblock-rtcp-xr-quality-monitoring-06 as a WG item. 
>>> 
>>> 2. Not ready to adopt.
>>>  * Would be very helpful if you provide your reasonings.
>>> 
>>> Regards
>>> Shida (as chair)
>>> _______________________________________________
>>> xrblock mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/xrblock
>> 
>> 
>> -- 
>> Colin Perkins
>> http://csperkins.org/
>> 
>> 
>> 
>> _______________________________________________
>> xrblock mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/xrblock
> _______________________________________________
> xrblock mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/xrblock



-- 
Colin Perkins
http://csperkins.org/