Re: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt

[Alan Clark <[e-mail]>]

Hi Qin

The idea of "reference" is incorporated into subjective test methodologies
and into specific subject test experiments. For example when a codec is
evaluated it is typically compared against other codecs with the same sample
rate (i.e. a narrowband codec is compared against other narrowband codecs) -
this means that the resulting MOS scores have a narrowband reference.

I am not aware of a specific standard related to "reference" and "scaling"
other than the definition of the subjective test methodologies that define
anchor conditions (e.g. MNRU's).

The same general idea applies to video - video on a mobile handset would
typically be evaluated against other mobile handsets - so reference/scaling
may apply to image resolution, screen size or both. Telchemy's technology
uses the terms Absolute and Relative MOS-V, we have described this in
contributions to ITU-T SG12 however nothing has been incorporated into a
standard as yet.

Alan


On 8/27/12 10:52 PM, "Qin Wu" <[e-mail]> wrote:

> Hi,
> Does "the reference" you refer to for the MoS definitions apply to video?
> Are there any standards we can refer to for the concept of "reference"
> and "scaling"?
> 
> Regards!
> -Qin
> ----- Original Message -----
> From: "Alan Clark" <[e-mail]>
> To: "Dan (Dan)" <[e-mail]>; "Qin Wu" <[e-mail]>;
> <[e-mail]>; "Al Morton" <[e-mail]>
> Sent: Tuesday, August 28, 2012 1:03 AM
> Subject: Re: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt
> 
> 
>> Hi Dan
>> 
>> A registry would be needed for the MOS definition - not for the codec type.
>> The MOS definitions would need to include both the reference and the scale,
>> for example:
>> 
>> 1. Reference:
>> - Narrowband (8kHz) speech
>> - Wideband (16kHz) speech
>> -....
>> 
>> Note that you can represent a Narrowband codec on both Narrowband and
>> Wideband scales but a Wideband codec has to be represented on a Wideband
>> scale.
>> 
>> Current thought within the industry is to map Superwideband and Fullband
>> codecs onto the Wideband scale however if this changes then there may be
>> additional references needed
>> 
>> For video - the issue of MOS reference is still a discussion item.
>> 
>> 2. Scaling:
>> This is a slightly more complicated issue. G.107 was developed in the mid
>> 1990's based on subjective test data from the 1980's and early 90's; a
>> "good" G.711 call would have a MOS of 4.45 as calculated by G.107.  If you
>> look at subjective test data from the later 1990's and 2000's you would find
>> that G.711 MOS scores tend to be around 4.1-4.2.  This may be due to
>> subjective test reference condition changes or possibly to listeners being
>> more used to digital audio and hence more critical.  Also the Japanese
>> national standard uses a lower MOS scaling that would give a "good" G.711
>> score of 3.8.
>> Telchemy uses the terms "ITU Scaled" to mean the scaling used for G.107,
>> "ACR Scaled" to mean - consistent with typical subjective test data, and
>> "Japanese TTC Scaled" to mean - consistent with the Japanese national
>> standard.  All of our customers seem to use the ACR Scaled range as this
>> aligns with typical quoted MOS scores for codecs. I'm not aware of ITU-T
>> SG12 having worked on this specific topic of scaling, although they are of
>> course very aware of the subject matter.
>> 
>> It would of course be possible to include a reference MOS in the QoE report
>> as an alternative to a more complex registry.
>> 
>> Best Regards
>> 
>> Alan
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On 8/27/12 10:21 AM, "Dan (Dan)" <[e-mail]> wrote:
>> 
>>> Hi Alan,
>>> 
>>> In the meeting in Vancouver a proposal was made to create a IANA
>>> registry for the calculation algorithms. Assuming that we go this way
>>> (starting from Al's initial list or something derived from it) do you
>>> believe that we need a second registry for the codec type?
>>> 
>>> Dan
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: [e-mail] [mailto:[e-mail]] On
>>>> Behalf Of Alan Clark
>>>> Sent: Thursday, August 16, 2012 5:59 PM
>>>> To: Qin Wu; [e-mail]; Al Morton
>>>> Subject: Re: [xrblock] comment on
>>> draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>>> 
>>>> 
>>>> Some further work is definitely needed on this draft with regard to
>>> the
>>>> MOS terminology. The various calculation methods for MOS either
>>> already
>>>> do, or soon will, provide MOS values for narrowband, wideband, super
>>>> wideband and fullband codecs; these MOS values occupy the same range
>>> and
>>>> hence the QoE block needs to indicate what the MOS reference is.
>>>> 
>>>> In addition there are national standards (e.g. Japan) that use a
>>>> different MOS scaling.
>>>> 
>>>> Alan
>>>> 
>>>> 
>>>> 
>>>> On 8/15/12 10:33 PM, "Qin Wu" <[e-mail]> wrote:
>>>> 
>>>>> Hi, Al:
>>>>> Thank for your general comments, I take some time to look at the
>>>>> standards for calculation again.
>>>>> please see my replies inline.
>>>>> 
>>>>> Regards!
>>>>> -Qin
>>>>> ----- Original Message -----
>>>>> From: "Al Morton" <[e-mail]>
>>>>> To: <[e-mail]>
>>>>> Sent: Thursday, August 02, 2012 5:01 AM
>>>>> Subject: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>>>> 
>>>>> 
>>>>>> Regarding:
>>>>>> http://tools.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>>>>> 
>>>>>>>    Calculation Algorithm (CALg):3 bits
>>>>>>> 
>>>>>>>          000 - ITU-T P.564 Compliant Algorithm [P.564] (Voice)
>>>>>>>          001 - G.107 [G.107] (Voice)
>>>>>>>          010 - ETSI TS 101 329-5 Annex E [ ETSI] (Voice)
>>>>>>>          011 - ITU-T P.NAMS [P.NAMS] (Multimedia)
>>>>>>>          100 - ITU-T P.NBAMS [P.NBAMS] (Multimedia)
>>>>>>>          101~111 - Reserved for future extension.
>>>>>>> 
>>>>>>>       G.107 and P.564 and ETSI TS101 329-5 specify three
>>> Calculation
>>>>>>>       algorithms or MoS algorithms that are used to estimate
>>> speech
>>>>>>>       quality or conversation quality.  P.NAMS and P.NBAMS specify
>>>> two
>>>>>>>       MoS algorithms that are used to estimate multimedia quality
>>>>>>>       including video quality, audio quality and audio-video
>>>> quality.
>>>>>> 
>>>>>> Specifying the standard for the calculation is a good start, but
>>>>>> these standards all have options and input parameters that must
>>> also
>>>>>> be specified in order to know what the MOS means.
>>>>> 
>>>>>> For example, we need to know what codec was used, because G.726
>>>>>> carries an equipment Impairment factor that limits the upper bound
>>> on
>>>>>> MOS, while G.711 does not.
>>>>> 
>>>>> [Qin]: I agree MoS value depends on the codec that is in use. That's
>>>>> why, in the current draft, We use 7-bit payload Type in the metric
>>>>> block to signal what codec or payload format is  in use for
>>> reporting
>>>>> interval.
>>>>> Please see the definition of Payload Type (PT) in the section 3.2.1.
>>>>> 
>>>>>> 
>>>>>> Some of the input parameters will be assumed (e.g., at default
>>>>>> values) while others will be measured, and the distinction between
>>>>>> measured and assumed parameters should also be made.
>>>>> 
>>>>> [Qin]:
>>>>> The input parameters may come from Delay metric block, Discard
>>> metric
>>>>> block, Burst Gap metrics block, JB metrics block, loss concealment
>>>>> metric block, concelament seconds metrics block, these metrics
>>> blocks
>>>>> are defined in other XR Block drafts by XRBLOCK WG .
>>>>> 
>>>>> In QoE metric block, 4 or 5 pameters are defined, i.e., a.Segment
>>> Type
>>>>> b.MoS Type c.MoS algorithm d. MoS value In these parameters, a~c can
>>>>> be counted as default values while d is counted as measured value.
>>>>> The other default values rely on specific computation model defined
>>> in
>>>>> each standard, e.g., P.564, G.104, P.NAMS.
>>>>> Assume each standard only provide one algorithm or one model, we can
>>>>> use  the standards name and payload type(i.e.,codec) to identify
>>> each
>>>>> MoS algorithm used or each MoS model used.
>>>>> 
>>>>>> As I mentioned, metrics with many optional parameters are somewhat
>>>>>> difficult to identify in a simple way, as we found with the IPPM
>>>>>> Metrics Registry (which we withdrew when we found it was
>>> insufficient
>>>>>> to describe the measurement results in an exact way).
>>>>>> 
>>>>>> Colin Perkins suggested the possibility a registry for the
>>>> calculation alg.
>>>>>> A registry could provide a single value index to a complicated set
>>> of
>>>>>> input parameters and other assumptions, and this might work for
>>> you.
>>>>> 
>>>>> [Qin]: Yes,depends on the input parameters and option we got, we can
>>>>> calculate the different MoS values however the computation model is
>>>>> same and will not be affected by the input parameters and options.
>>>>> So maybe rather than registering MoS algorithm, we should register
>>>>> assessment model or computation model.
>>>>> We can use computation model defined in the standard and payload
>>> type
>>>>> as single value index.
>>>>> 
>>>>>> hope this helps,
>>>>>> Al
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> xrblock mailing list
>>>>>> [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/xrblock
>>>>>> 
>>>>> _______________________________________________
>>>>> xrblock mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/xrblock
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> xrblock mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/xrblock
>> 
>> 
>> 
>> _______________________________________________
>> xrblock mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/xrblock
>>