Re: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt

[Qin Wu <[e-mail]>]

Hi,Alan:
Sorry for late reply and finally get back to this topic.
----- Original Message ----- 
From: "Alan Clark" <[e-mail]>
To: "Qin Wu" <[e-mail]>; "Dan (Dan)" <[e-mail]>; <[e-mail]>; "Al Morton" <[e-mail]>
Sent: Tuesday, August 28, 2012 9:17 PM
Subject: Re: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt


> Hi Qin
> 
> The idea of "reference" is incorporated into subjective test methodologies
> and into specific subject test experiments. For example when a codec is
> evaluated it is typically compared against other codecs with the same sample
> rate (i.e. a narrowband codec is compared against other narrowband codecs) -
> this means that the resulting MOS scores have a narrowband reference.
> 
> I am not aware of a specific standard related to "reference" and "scaling"
> other than the definition of the subjective test methodologies that define
> anchor conditions (e.g. MNRU's).

[Qin]: If we consider to add two new defintion in the protocol format for "MoS reference"
and "MoS scaling", what it will be look like? What's your proposed text?

> The same general idea applies to video - video on a mobile handset would
> typically be evaluated against other mobile handsets - so reference/scaling
> may apply to image resolution, screen size or both. Telchemy's technology
> uses the terms Absolute and Relative MOS-V, we have described this in
> contributions to ITU-T SG12 however nothing has been incorporated into a
> standard as yet.

[Qin]: It seems you mixed reference and scaling for video. I think for video,
the reference should apply to image resolution/screen size, e.g., High definition, Standard Defintion.
But I am not sure how scaling apply to image resolution as well. What am I missing?

> Alan
> 
> 
> On 8/27/12 10:52 PM, "Qin Wu" <[e-mail]> wrote:
> 
>> Hi,
>> Does "the reference" you refer to for the MoS definitions apply to video?
>> Are there any standards we can refer to for the concept of "reference"
>> and "scaling"?
>> 
>> Regards!
>> -Qin
>> ----- Original Message -----
>> From: "Alan Clark" <[e-mail]>
>> To: "Dan (Dan)" <[e-mail]>; "Qin Wu" <[e-mail]>;
>> <[e-mail]>; "Al Morton" <[e-mail]>
>> Sent: Tuesday, August 28, 2012 1:03 AM
>> Subject: Re: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>> 
>> 
>>> Hi Dan
>>> 
>>> A registry would be needed for the MOS definition - not for the codec type.
>>> The MOS definitions would need to include both the reference and the scale,
>>> for example:
>>> 
>>> 1. Reference:
>>> - Narrowband (8kHz) speech
>>> - Wideband (16kHz) speech
>>> -....
>>> 
>>> Note that you can represent a Narrowband codec on both Narrowband and
>>> Wideband scales but a Wideband codec has to be represented on a Wideband
>>> scale.
>>> 
>>> Current thought within the industry is to map Superwideband and Fullband
>>> codecs onto the Wideband scale however if this changes then there may be
>>> additional references needed
>>> 
>>> For video - the issue of MOS reference is still a discussion item.
>>> 
>>> 2. Scaling:
>>> This is a slightly more complicated issue. G.107 was developed in the mid
>>> 1990's based on subjective test data from the 1980's and early 90's; a
>>> "good" G.711 call would have a MOS of 4.45 as calculated by G.107.  If you
>>> look at subjective test data from the later 1990's and 2000's you would find
>>> that G.711 MOS scores tend to be around 4.1-4.2.  This may be due to
>>> subjective test reference condition changes or possibly to listeners being
>>> more used to digital audio and hence more critical.  Also the Japanese
>>> national standard uses a lower MOS scaling that would give a "good" G.711
>>> score of 3.8.
>>> Telchemy uses the terms "ITU Scaled" to mean the scaling used for G.107,
>>> "ACR Scaled" to mean - consistent with typical subjective test data, and
>>> "Japanese TTC Scaled" to mean - consistent with the Japanese national
>>> standard.  All of our customers seem to use the ACR Scaled range as this
>>> aligns with typical quoted MOS scores for codecs. I'm not aware of ITU-T
>>> SG12 having worked on this specific topic of scaling, although they are of
>>> course very aware of the subject matter.
>>> 
>>> It would of course be possible to include a reference MOS in the QoE report
>>> as an alternative to a more complex registry.
>>> 
>>> Best Regards
>>> 
>>> Alan
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 8/27/12 10:21 AM, "Dan (Dan)" <[e-mail]> wrote:
>>> 
>>>> Hi Alan,
>>>> 
>>>> In the meeting in Vancouver a proposal was made to create a IANA
>>>> registry for the calculation algorithms. Assuming that we go this way
>>>> (starting from Al's initial list or something derived from it) do you
>>>> believe that we need a second registry for the codec type?
>>>> 
>>>> Dan
>>>> 
>>>> 
>>>>> -----Original Message-----
>>>>> From: [e-mail] [mailto:[e-mail]] On
>>>>> Behalf Of Alan Clark
>>>>> Sent: Thursday, August 16, 2012 5:59 PM
>>>>> To: Qin Wu; [e-mail]; Al Morton
>>>>> Subject: Re: [xrblock] comment on
>>>> draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>>>> 
>>>>> 
>>>>> Some further work is definitely needed on this draft with regard to
>>>> the
>>>>> MOS terminology. The various calculation methods for MOS either
>>>> already
>>>>> do, or soon will, provide MOS values for narrowband, wideband, super
>>>>> wideband and fullband codecs; these MOS values occupy the same range
>>>> and
>>>>> hence the QoE block needs to indicate what the MOS reference is.
>>>>> 
>>>>> In addition there are national standards (e.g. Japan) that use a
>>>>> different MOS scaling.
>>>>> 
>>>>> Alan
>>>>> 
>>>>> 
>>>>> 
>>>>> On 8/15/12 10:33 PM, "Qin Wu" <[e-mail]> wrote:
>>>>> 
>>>>>> Hi, Al:
>>>>>> Thank for your general comments, I take some time to look at the
>>>>>> standards for calculation again.
>>>>>> please see my replies inline.
>>>>>> 
>>>>>> Regards!
>>>>>> -Qin
>>>>>> ----- Original Message -----
>>>>>> From: "Al Morton" <[e-mail]>
>>>>>> To: <[e-mail]>
>>>>>> Sent: Thursday, August 02, 2012 5:01 AM
>>>>>> Subject: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>>>>> 
>>>>>> 
>>>>>>> Regarding:
>>>>>>> http://tools.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>>>>>> 
>>>>>>>>    Calculation Algorithm (CALg):3 bits
>>>>>>>> 
>>>>>>>>          000 - ITU-T P.564 Compliant Algorithm [P.564] (Voice)
>>>>>>>>          001 - G.107 [G.107] (Voice)
>>>>>>>>          010 - ETSI TS 101 329-5 Annex E [ ETSI] (Voice)
>>>>>>>>          011 - ITU-T P.NAMS [P.NAMS] (Multimedia)
>>>>>>>>          100 - ITU-T P.NBAMS [P.NBAMS] (Multimedia)
>>>>>>>>          101~111 - Reserved for future extension.
>>>>>>>> 
>>>>>>>>       G.107 and P.564 and ETSI TS101 329-5 specify three
>>>> Calculation
>>>>>>>>       algorithms or MoS algorithms that are used to estimate
>>>> speech
>>>>>>>>       quality or conversation quality.  P.NAMS and P.NBAMS specify
>>>>> two
>>>>>>>>       MoS algorithms that are used to estimate multimedia quality
>>>>>>>>       including video quality, audio quality and audio-video
>>>>> quality.
>>>>>>> 
>>>>>>> Specifying the standard for the calculation is a good start, but
>>>>>>> these standards all have options and input parameters that must
>>>> also
>>>>>>> be specified in order to know what the MOS means.
>>>>>> 
>>>>>>> For example, we need to know what codec was used, because G.726
>>>>>>> carries an equipment Impairment factor that limits the upper bound
>>>> on
>>>>>>> MOS, while G.711 does not.
>>>>>> 
>>>>>> [Qin]: I agree MoS value depends on the codec that is in use. That's
>>>>>> why, in the current draft, We use 7-bit payload Type in the metric
>>>>>> block to signal what codec or payload format is  in use for
>>>> reporting
>>>>>> interval.
>>>>>> Please see the definition of Payload Type (PT) in the section 3.2.1.
>>>>>> 
>>>>>>> 
>>>>>>> Some of the input parameters will be assumed (e.g., at default
>>>>>>> values) while others will be measured, and the distinction between
>>>>>>> measured and assumed parameters should also be made.
>>>>>> 
>>>>>> [Qin]:
>>>>>> The input parameters may come from Delay metric block, Discard
>>>> metric
>>>>>> block, Burst Gap metrics block, JB metrics block, loss concealment
>>>>>> metric block, concelament seconds metrics block, these metrics
>>>> blocks
>>>>>> are defined in other XR Block drafts by XRBLOCK WG .
>>>>>> 
>>>>>> In QoE metric block, 4 or 5 pameters are defined, i.e., a.Segment
>>>> Type
>>>>>> b.MoS Type c.MoS algorithm d. MoS value In these parameters, a~c can
>>>>>> be counted as default values while d is counted as measured value.
>>>>>> The other default values rely on specific computation model defined
>>>> in
>>>>>> each standard, e.g., P.564, G.104, P.NAMS.
>>>>>> Assume each standard only provide one algorithm or one model, we can
>>>>>> use  the standards name and payload type(i.e.,codec) to identify
>>>> each
>>>>>> MoS algorithm used or each MoS model used.
>>>>>> 
>>>>>>> As I mentioned, metrics with many optional parameters are somewhat
>>>>>>> difficult to identify in a simple way, as we found with the IPPM
>>>>>>> Metrics Registry (which we withdrew when we found it was
>>>> insufficient
>>>>>>> to describe the measurement results in an exact way).
>>>>>>> 
>>>>>>> Colin Perkins suggested the possibility a registry for the
>>>>> calculation alg.
>>>>>>> A registry could provide a single value index to a complicated set
>>>> of
>>>>>>> input parameters and other assumptions, and this might work for
>>>> you.
>>>>>> 
>>>>>> [Qin]: Yes,depends on the input parameters and option we got, we can
>>>>>> calculate the different MoS values however the computation model is
>>>>>> same and will not be affected by the input parameters and options.
>>>>>> So maybe rather than registering MoS algorithm, we should register
>>>>>> assessment model or computation model.
>>>>>> We can use computation model defined in the standard and payload
>>>> type
>>>>>> as single value index.
>>>>>> 
>>>>>>> hope this helps,
>>>>>>> Al
>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> xrblock mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/xrblock
>>>>>>> 
>>>>>> _______________________________________________
>>>>>> xrblock mailing list
>>>>>> [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/xrblock
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> xrblock mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/xrblock
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> xrblock mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/xrblock
>>> 
> 
> 
> 
>