Re: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt

[Alan Clark <[e-mail]>]

> Hi,Alan:
> Sorry for late reply and finally get back to this topic.
> ----- Original Message -----
> From: "Alan Clark" <[e-mail]>
> To: "Qin Wu" <[e-mail]>; "Dan (Dan)" <[e-mail]>;
> <[e-mail]>; "Al Morton" <[e-mail]>
> Sent: Tuesday, August 28, 2012 9:17 PM
> Subject: Re: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt
> 
> 
>> Hi Qin
>> 
>> The idea of "reference" is incorporated into subjective test methodologies
>> and into specific subject test experiments. For example when a codec is
>> evaluated it is typically compared against other codecs with the same sample
>> rate (i.e. a narrowband codec is compared against other narrowband codecs) -
>> this means that the resulting MOS scores have a narrowband reference.
>> 
>> I am not aware of a specific standard related to "reference" and "scaling"
>> other than the definition of the subjective test methodologies that define
>> anchor conditions (e.g. MNRU's).
> 
> [Qin]: If we consider to add two new defintion in the protocol format for "MoS
> reference"
> and "MoS scaling", what it will be look like? What's your proposed text?

The best approach would be to have the concept of MOS Reference - initially
defining this only for Speech as it is not yet defined for Audio or Video.
The MOS Reference for Speech would be Narrowband (3.4kHz) or Wideband
(7kHz), where the frequency refers to the audio bandwidth and not the sample
rate (e.g. an 8kHz sample rate is used on traditional telecom networks and
audio bandwidth is filtered to 3.4kz).

MOS Scaling is a term that Telchemy uses explicitly however the concept is
implicitly used within the industry when quoting MOS scores for codecs and
measurements. The only standardized reference that I'm aware of is the
Japanese TTC scaling.

> 
>> The same general idea applies to video - video on a mobile handset would
>> typically be evaluated against other mobile handsets - so reference/scaling
>> may apply to image resolution, screen size or both. Telchemy's technology
>> uses the terms Absolute and Relative MOS-V, we have described this in
>> contributions to ITU-T SG12 however nothing has been incorporated into a
>> standard as yet.
> 
> [Qin]: It seems you mixed reference and scaling for video. I think for video,
> the reference should apply to image resolution/screen size, e.g., High
> definition, Standard Defintion.
> But I am not sure how scaling apply to image resolution as well. What am I
> missing?
> 
In many cases (e.g. middle of the network) the screen size is not known
although the image resolution is. Taking screen size into account is valid
as a factor in determining a subject score however has the unfortunate
side-effect of making the same video stream with the same encoding/
bandwidth/ impairments/ resolution/ content ... have different MOS scores
depending on the playout device size. This is "correct" however can be
confusing for network operations engineers.

Alan

>> Alan
>> 
>> 
>> On 8/27/12 10:52 PM, "Qin Wu" <[e-mail]> wrote:
>> 
>>> Hi,
>>> Does "the reference" you refer to for the MoS definitions apply to video?
>>> Are there any standards we can refer to for the concept of "reference"
>>> and "scaling"?
>>> 
>>> Regards!
>>> -Qin
>>> ----- Original Message -----
>>> From: "Alan Clark" <[e-mail]>
>>> To: "Dan (Dan)" <[e-mail]>; "Qin Wu" <[e-mail]>;
>>> <[e-mail]>; "Al Morton" <[e-mail]>
>>> Sent: Tuesday, August 28, 2012 1:03 AM
>>> Subject: Re: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>> 
>>> 
>>>> Hi Dan
>>>> 
>>>> A registry would be needed for the MOS definition - not for the codec type.
>>>> The MOS definitions would need to include both the reference and the scale,
>>>> for example:
>>>> 
>>>> 1. Reference:
>>>> - Narrowband (8kHz) speech
>>>> - Wideband (16kHz) speech
>>>> -....
>>>> 
>>>> Note that you can represent a Narrowband codec on both Narrowband and
>>>> Wideband scales but a Wideband codec has to be represented on a Wideband
>>>> scale.
>>>> 
>>>> Current thought within the industry is to map Superwideband and Fullband
>>>> codecs onto the Wideband scale however if this changes then there may be
>>>> additional references needed
>>>> 
>>>> For video - the issue of MOS reference is still a discussion item.
>>>> 
>>>> 2. Scaling:
>>>> This is a slightly more complicated issue. G.107 was developed in the mid
>>>> 1990's based on subjective test data from the 1980's and early 90's; a
>>>> "good" G.711 call would have a MOS of 4.45 as calculated by G.107.  If you
>>>> look at subjective test data from the later 1990's and 2000's you would
>>>> find
>>>> that G.711 MOS scores tend to be around 4.1-4.2.  This may be due to
>>>> subjective test reference condition changes or possibly to listeners being
>>>> more used to digital audio and hence more critical.  Also the Japanese
>>>> national standard uses a lower MOS scaling that would give a "good" G.711
>>>> score of 3.8.
>>>> Telchemy uses the terms "ITU Scaled" to mean the scaling used for G.107,
>>>> "ACR Scaled" to mean - consistent with typical subjective test data, and
>>>> "Japanese TTC Scaled" to mean - consistent with the Japanese national
>>>> standard.  All of our customers seem to use the ACR Scaled range as this
>>>> aligns with typical quoted MOS scores for codecs. I'm not aware of ITU-T
>>>> SG12 having worked on this specific topic of scaling, although they are of
>>>> course very aware of the subject matter.
>>>> 
>>>> It would of course be possible to include a reference MOS in the QoE report
>>>> as an alternative to a more complex registry.
>>>> 
>>>> Best Regards
>>>> 
>>>> Alan
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 8/27/12 10:21 AM, "Dan (Dan)" <[e-mail]> wrote:
>>>> 
>>>>> Hi Alan,
>>>>> 
>>>>> In the meeting in Vancouver a proposal was made to create a IANA
>>>>> registry for the calculation algorithms. Assuming that we go this way
>>>>> (starting from Al's initial list or something derived from it) do you
>>>>> believe that we need a second registry for the codec type?
>>>>> 
>>>>> Dan
>>>>> 
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: [e-mail] [mailto:[e-mail]] On
>>>>>> Behalf Of Alan Clark
>>>>>> Sent: Thursday, August 16, 2012 5:59 PM
>>>>>> To: Qin Wu; [e-mail]; Al Morton
>>>>>> Subject: Re: [xrblock] comment on
>>>>> draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>>>>> 
>>>>>> 
>>>>>> Some further work is definitely needed on this draft with regard to
>>>>> the
>>>>>> MOS terminology. The various calculation methods for MOS either
>>>>> already
>>>>>> do, or soon will, provide MOS values for narrowband, wideband, super
>>>>>> wideband and fullband codecs; these MOS values occupy the same range
>>>>> and
>>>>>> hence the QoE block needs to indicate what the MOS reference is.
>>>>>> 
>>>>>> In addition there are national standards (e.g. Japan) that use a
>>>>>> different MOS scaling.
>>>>>> 
>>>>>> Alan
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 8/15/12 10:33 PM, "Qin Wu" <[e-mail]> wrote:
>>>>>> 
>>>>>>> Hi, Al:
>>>>>>> Thank for your general comments, I take some time to look at the
>>>>>>> standards for calculation again.
>>>>>>> please see my replies inline.
>>>>>>> 
>>>>>>> Regards!
>>>>>>> -Qin
>>>>>>> ----- Original Message -----
>>>>>>> From: "Al Morton" <[e-mail]>
>>>>>>> To: <[e-mail]>
>>>>>>> Sent: Thursday, August 02, 2012 5:01 AM
>>>>>>> Subject: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>>>>>> 
>>>>>>> 
>>>>>>>> Regarding:
>>>>>>>> http://tools.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>>>>>>> 
>>>>>>>>>    Calculation Algorithm (CALg):3 bits
>>>>>>>>> 
>>>>>>>>>          000 - ITU-T P.564 Compliant Algorithm [P.564] (Voice)
>>>>>>>>>          001 - G.107 [G.107] (Voice)
>>>>>>>>>          010 - ETSI TS 101 329-5 Annex E [ ETSI] (Voice)
>>>>>>>>>          011 - ITU-T P.NAMS [P.NAMS] (Multimedia)
>>>>>>>>>          100 - ITU-T P.NBAMS [P.NBAMS] (Multimedia)
>>>>>>>>>          101~111 - Reserved for future extension.
>>>>>>>>> 
>>>>>>>>>       G.107 and P.564 and ETSI TS101 329-5 specify three
>>>>> Calculation
>>>>>>>>>       algorithms or MoS algorithms that are used to estimate
>>>>> speech
>>>>>>>>>       quality or conversation quality.  P.NAMS and P.NBAMS specify
>>>>>> two
>>>>>>>>>       MoS algorithms that are used to estimate multimedia quality
>>>>>>>>>       including video quality, audio quality and audio-video
>>>>>> quality.
>>>>>>>> 
>>>>>>>> Specifying the standard for the calculation is a good start, but
>>>>>>>> these standards all have options and input parameters that must
>>>>> also
>>>>>>>> be specified in order to know what the MOS means.
>>>>>>> 
>>>>>>>> For example, we need to know what codec was used, because G.726
>>>>>>>> carries an equipment Impairment factor that limits the upper bound
>>>>> on
>>>>>>>> MOS, while G.711 does not.
>>>>>>> 
>>>>>>> [Qin]: I agree MoS value depends on the codec that is in use. That's
>>>>>>> why, in the current draft, We use 7-bit payload Type in the metric
>>>>>>> block to signal what codec or payload format is  in use for
>>>>> reporting
>>>>>>> interval.
>>>>>>> Please see the definition of Payload Type (PT) in the section 3.2.1.
>>>>>>> 
>>>>>>>> 
>>>>>>>> Some of the input parameters will be assumed (e.g., at default
>>>>>>>> values) while others will be measured, and the distinction between
>>>>>>>> measured and assumed parameters should also be made.
>>>>>>> 
>>>>>>> [Qin]:
>>>>>>> The input parameters may come from Delay metric block, Discard
>>>>> metric
>>>>>>> block, Burst Gap metrics block, JB metrics block, loss concealment
>>>>>>> metric block, concelament seconds metrics block, these metrics
>>>>> blocks
>>>>>>> are defined in other XR Block drafts by XRBLOCK WG .
>>>>>>> 
>>>>>>> In QoE metric block, 4 or 5 pameters are defined, i.e., a.Segment
>>>>> Type
>>>>>>> b.MoS Type c.MoS algorithm d. MoS value In these parameters, a~c can
>>>>>>> be counted as default values while d is counted as measured value.
>>>>>>> The other default values rely on specific computation model defined
>>>>> in
>>>>>>> each standard, e.g., P.564, G.104, P.NAMS.
>>>>>>> Assume each standard only provide one algorithm or one model, we can
>>>>>>> use  the standards name and payload type(i.e.,codec) to identify
>>>>> each
>>>>>>> MoS algorithm used or each MoS model used.
>>>>>>> 
>>>>>>>> As I mentioned, metrics with many optional parameters are somewhat
>>>>>>>> difficult to identify in a simple way, as we found with the IPPM
>>>>>>>> Metrics Registry (which we withdrew when we found it was
>>>>> insufficient
>>>>>>>> to describe the measurement results in an exact way).
>>>>>>>> 
>>>>>>>> Colin Perkins suggested the possibility a registry for the
>>>>>> calculation alg.
>>>>>>>> A registry could provide a single value index to a complicated set
>>>>> of
>>>>>>>> input parameters and other assumptions, and this might work for
>>>>> you.
>>>>>>> 
>>>>>>> [Qin]: Yes,depends on the input parameters and option we got, we can
>>>>>>> calculate the different MoS values however the computation model is
>>>>>>> same and will not be affected by the input parameters and options.
>>>>>>> So maybe rather than registering MoS algorithm, we should register
>>>>>>> assessment model or computation model.
>>>>>>> We can use computation model defined in the standard and payload
>>>>> type
>>>>>>> as single value index.
>>>>>>> 
>>>>>>>> hope this helps,
>>>>>>>> Al
>>>>>>>> 
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> xrblock mailing list
>>>>>>>> [e-mail]
>>>>>>>> https://www.ietf.org/mailman/listinfo/xrblock
>>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> xrblock mailing list
>>>>>>> [e-mail]
>>>>>>> https://www.ietf.org/mailman/listinfo/xrblock
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> xrblock mailing list
>>>>>> [e-mail]
>>>>>> https://www.ietf.org/mailman/listinfo/xrblock
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> xrblock mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/xrblock
>>>> 
>> 
>> 
>> 
>>