Re: [xrblock] Call for adopting xr-quality-monitoring-06 as WG item

[Qin Wu <[e-mail]>]

Hi, Colin and Chairs:
>----- Original Message ----- 
>From: "Colin Perkins" <[e-mail]>
>To: "Qin Wu" <[e-mail]>
>Cc: "Shida Schubert" <[e-mail]>; <[e-mail]>
>Sent: Wednesday, December 21, 2011 1:06 AM
>Subject: Re: [xrblock] Call for adopting xr-quality-monitoring-06 as WG item


>Qin,

>See inline.
>Colin


>>On 20 Dec 2011, at 01:54, Qin Wu wrote:
>> Hi, Colin:
>> I understand your concerns. It is painful, but that's the tradeoff we should made.
>> Please see my clarification belows.
>> ----- Original Message ----- 
>> From: "Colin Perkins" <[e-mail]>
>> To: "Shida Schubert" <[e-mail]>
>> Cc: <[e-mail]>
>> Sent: Tuesday, December 20, 2011 1:13 AM
>> Subject: Re: [xrblock] Call for adopting xr-quality-monitoring-06 as WG item
>> 
>> 
>>> Shida,
>>> 
>>> I don't think this draft is ready to adopt:
>>> 
>>> - The single-stream metric mixes audio and video MOS into a single metric block, with a grab-bag of MOS type and calculation algorithm parameters. It seems that little thought has gone into designing >>>the XR block format to prohibit invalid combinations (e.g., MT = MOS-V with CALg = G.107), and there is no guidance on which algorithm is appropriate for which MOS type. 
>>> 
>> [Qin]:  We do have guidance to prevent such combination  in three places
>> a. See MoS Type (MT) field definition, it said:
> "
>>      If MoS type is MoS-LQ and MoS-CQ, the MoS value can be calculated
>>      based on ITU-T G.107[G.107], ITU-T P.564 [P.564]or ETSI TS 101
>>      329-5 [ETSI], if the Mos type is MoS-V or MoS-AV, the Mos value
>>      can be calculated based on ITU-T P.NAMS [P.NAMS]or ITU-T P.NBAMS
>>      [P.NBAMS]. 
>> "
>> b. Also see Mos Type (MT) field defintion, it said:
>> "
>>     If MoS Type does not match the MoS algorithm in the report (e.g.,
>>      specify a voice MOS algorithm for a video quality MOS), the
>>      receiver should also discard this report.
>> "
>> c.  See Calculation Algorithm (CALg) field definition
>> It distinguish Audio MoS algorithm from video MoS algorithm by adding "(voice)" into Audio MoS algorithm.

>I agree that there is guidance to avoid invalid combinations. However, a better design would prevent invalid combinations by appropriate choice of report block structure. 

>I still see no guidance on which algorithm is appropriate for which MOS type.

[Qin]: I get your point. In the current draft, we list 5 MoS algorithms, 3 MOS algorithms specifed in the G.107 and P.564 and ETSI TS101 329-5 are only used to estimate speech quality or conversation quality. 2 MoS algorithms specified in the P.NAMS and P.NBAMS can be used to estimate either audio quality or video quality or multimedia quality. Also I like to propose to add one MoS Type MoS-A into the draft to distinguish conversation MoS and speech MoS from audio MoS.  

If you think this clarificatin does address your comments, I can put them into the update of the draft.

>>> The format seems to follow the RFC 3611 model, throwing together a set of unrelated parameters, rather than the approach documents in the monarch draft.
>> 
>> [Qin]: In this draft, we are not allowed to put "single stream segment","multi-layer segment" and "multi-channel segment" into one XR Block Report. please see the note in the segement i field definition
>> it said:
>> "
>>      Note that in these
>>      three segment type,any two segment types can not be present in the
>>      same metric block.
>> "
>> That is to say if we choose to use "single stream segment" to report MoS, we will not use "multi-channel segment" or "multi-layer segment" . If we choose to use multi-channel segment, we will not use 
>> "single stream segment" or "multi-layer segment".
>> 
>> Also we may need to measure MoS-CQ using three different audio MoS algorithms, e.g., 
>> Using ITU-T P.564 to calculate MoS-CQ1
>> Using G.107 to calculate MoS-CQ2
>> Using ETSI TS 101 329-5 to calculate MoS-CQ3
>> 
>> In this case,  MoS-CQ1 and MoS-CQ2 and MoS-CQ3 are all MoS value, but using different audio MoS algorithm, so the question is how to report them, we have two options:
>> 1. Report each MoS-CQ value carrried in the "single stream segment" using three XR Blocks 
>> 2. Report three MoS-CQ vlaues carried in three "single stream segment" using only one XR Block.
>> I think the option 2 is a better approach, this approach only report MoS value related to one single MoS Type and doesn't put a set of unrelated parameters together and therefore doesn't conflict the rule >>we set in MONARCH.
>> However if we also measure MoS-LQ using three different audio MoS algorithms and report MoS-CQ value and MoS-LQ value carried in two "single stream segment" using one single XR Block, that >>does mean "put a set of unrelated parameters together" and therefore conflict with Monarch.
>> 
>> So my suggestion is to add a sentence to say only one MoS type is allowed to report in one single XR Block.
>> 
>> If you agree with this, I can revise the draft to make this clear. 

>This change doesn't address my concern, which is that a single draft is describing three different types of report block. The monarch draft was intended to recommend different report blocks be developed >in different drafts, so they can be implemented separately, in a mix-and-match style, rather than being combined into a single draft.

[Qin]: I agree, I have no strong opinon on this. I am okay to either put them toether in one single draft or document them separately.
but my difficulty is these three type of report blocks all report MoS value and are all for QoE metric reporting. I am not worried about block type space but I do worry 
about or need to figure out how to distinguish one another. But if we put them together, we don't have such issue. Also it is not difficult for the user to place his order by choosing different segment type
rather than place his order by choosing different block type for the same purpose(QoE metric reporting).

>>> - The multi-layer segment is ill-specified: does it support audio only, video only, or both depending on the Media Type bit? 
>> 
>> [Qin]: multi-layer segment does support both video and audio, 

>None of the field descriptions in Section 3.2.2 mention audio; they're all video only.

[Qin]: You are right, I check RFC6190, draft-ietf-avt-rtp-mvc-01, it looks to me these specification only support video. Audio seems to be carried in the different RTP session or different
RTP stream with different SSRC. And then using CNAME to group audio with layered video. Am I right?
If yes, can we say multi-layer segment only support video?

>>> Which calculation algorithm makes sense here, since there is no guidance given, and it's possible to specify any algorithm? 
>>> 
>> [Qin]: currently we only allow 5 MoS algorithms.since three Audio MoS algorithms have already been standardized and the other two are in standardization progress.

>This doest address my question. What algorithms make sense in the context of layered media?

[Qin]: Do we really have layered audio using RTP payload format?
If no, I think P.NAMS or PNBAMS is sufficient enough for layed media.

>>> What is the SSID field? How it is used? 
>> 
>> [Qin]:SSID is sub stream identifier and can be used to identifier each layer if all the layers are carried in the same RTP stream.

>More detail is needed in the draft.

[Qin]: Okay.

>>> It's not sufficient to simply say that a "NAL unit type is one example", the draft needs to give normative rules for the use of this field.
>> 
>> [Qin]:Agree.
>> 
>>> - The media type, MOS type, and calculation algorithm for the multi-channel segment are ill-specified in similar ways to the multi-layer segment.
>> 
>> [Qin]: Yes, you are right.
>> 
>>> Also, it's not clear that the channel mapping in RFC 3550 Section 4.1 is the only one in use.
>> 
>> [Qin]: Good point, but that's what we have in mind currently.
>> 
>>> - In general, it's not clear to me that including multiple different types of MOS report (single-stream, multi-layer, and multi-channel) in a single draft fits with the monitoring architecture, which encourages >small, single-use, XR block definitions. I would suggest that the three different MOS report types would be better defined as three drafts: there is little in common between them.
>> 
>> [Qin]: As I clarified above, these three MOS report types can not be put into one single XR Block.They are excluded between one another. 
>> But If WG think it is a good idea to merge them together, I am okay with that.

I agree that they should not be included in a single XR block. My question, however, was whether they should be in a single draft. 

[Qin]: See above, also I like to hear chairs' opinion.

>>> - The draft provides no meaningful guidelines for registration of new MOS algorithms, and has an inadequate IANA registry.
>> 
>> [Qin]: We do have in the (-06), section 5.4 of  xr-quality-monitoring-06

>Much more detail on the registration procedures, and guidelines for what should be registered, is needed.

[Qin]: Okay.

>> "
>> 5.4.  New registry of calculation algorithms
>> 
>>   This document creates a new registry to be called "RTCP XR QoE metric
>>   block - Calculation Algorithm" as a sub-registry of the "RTP Control
>>   Protocol Extended Reports (RTCP XR) Block Type Registry".  Policies
>>   for this new registry are as follows:
>> 
>>   o  Entries in the registry are integers.  The valid range is 0 to 7
>>      corresponding to the 3-bit field "CAlg" in the block.  Values are
>>      to be recorded in decimal.
>> 
>>   o  Initial assignments are as follows:
>> 
>>      1.  ITU-T P.564 Compliant Algorithm [P.564] (Voice)
>>      2.  G.107 [G.107] (Voice)
>>      3.  ETSI TS 101 329-5 Annex E ETSI] (Voice)
>>      4.  ITU-T P.NAMS [P.NAMS]
>>      5.  ITU-T P.NBAMS [P.NBAMS]
>> "
>>> Hope this makes sense.
>>> 
>>> Colin
>>> 
>>> 
>>> On 19 Dec 2011, at 11:55, Shida Schubert wrote:
>>>> This is a call to see whether people are happy 
>>>> with the latest xr-quality-monitoring-06 draft for addressing 
>>>> the milestone "QoE Metrics Reporting".
>>>> 
>>>> http://tools.ietf.org/html/draft-wu-xrblock-rtcp-xr-quality-monitoring-06
>>>> 
>>>> xr-que-00 which used to be an AVT draft also addresses 
>>>> the same milestone. 
>>>> 
>>>> http://tools.ietf.org/id/draft-clark-xrblock-rtcp-xr-qoe-00.txt
>>>> 
>>>> The xr-quality-monitoring-06 claims to merge the two 
>>>> into one. 
>>>> 
>>>> Please response to express your option for the followings. 
>>>> 
>>>> 1. Adopt the draft-wu-xrblock-rtcp-xr-quality-monitoring-06 as a WG item. 
>>>> 
>>>> 2. Not ready to adopt.
>>>>  * Would be very helpful if you provide your reasonings.
>>>> 
>>>> Regards
>>>> Shida (as chair)
>>> _______________________________________________
>>> xrblock mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/xrblock
>> 
>> 
>> -- 
>> Colin Perkins
>> http://csperkins.org/
>> 
>> 
>> 
>> _______________________________________________
>> xrblock mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/xrblock
> _______________________________________________
> xrblock mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/xrblock



-- 
Colin Perkins
http://csperkins.org/