Re: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt

[Alan Clark <[e-mail]>]

Hi Dan

A registry would be needed for the MOS definition - not for the codec type.
The MOS definitions would need to include both the reference and the scale,
for example:

1. Reference:
- Narrowband (8kHz) speech
- Wideband (16kHz) speech
-....

Note that you can represent a Narrowband codec on both Narrowband and
Wideband scales but a Wideband codec has to be represented on a Wideband
scale.

Current thought within the industry is to map Superwideband and Fullband
codecs onto the Wideband scale however if this changes then there may be
additional references needed

For video - the issue of MOS reference is still a discussion item.

2. Scaling:
This is a slightly more complicated issue. G.107 was developed in the mid
1990's based on subjective test data from the 1980's and early 90's; a
"good" G.711 call would have a MOS of 4.45 as calculated by G.107.  If you
look at subjective test data from the later 1990's and 2000's you would find
that G.711 MOS scores tend to be around 4.1-4.2.  This may be due to
subjective test reference condition changes or possibly to listeners being
more used to digital audio and hence more critical.  Also the Japanese
national standard uses a lower MOS scaling that would give a "good" G.711
score of 3.8.
Telchemy uses the terms "ITU Scaled" to mean the scaling used for G.107,
"ACR Scaled" to mean - consistent with typical subjective test data, and
"Japanese TTC Scaled" to mean - consistent with the Japanese national
standard.  All of our customers seem to use the ACR Scaled range as this
aligns with typical quoted MOS scores for codecs. I'm not aware of ITU-T
SG12 having worked on this specific topic of scaling, although they are of
course very aware of the subject matter.

It would of course be possible to include a reference MOS in the QoE report
as an alternative to a more complex registry.

Best Regards

Alan









On 8/27/12 10:21 AM, "Dan (Dan)" <[e-mail]> wrote:

> Hi Alan,
> 
> In the meeting in Vancouver a proposal was made to create a IANA
> registry for the calculation algorithms. Assuming that we go this way
> (starting from Al's initial list or something derived from it) do you
> believe that we need a second registry for the codec type?
> 
> Dan
> 
> 
>> -----Original Message-----
>> From: [e-mail] [mailto:[e-mail]] On
>> Behalf Of Alan Clark
>> Sent: Thursday, August 16, 2012 5:59 PM
>> To: Qin Wu; [e-mail]; Al Morton
>> Subject: Re: [xrblock] comment on
> draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>> 
>> 
>> Some further work is definitely needed on this draft with regard to
> the
>> MOS terminology. The various calculation methods for MOS either
> already
>> do, or soon will, provide MOS values for narrowband, wideband, super
>> wideband and fullband codecs; these MOS values occupy the same range
> and
>> hence the QoE block needs to indicate what the MOS reference is.
>> 
>> In addition there are national standards (e.g. Japan) that use a
>> different MOS scaling.
>> 
>> Alan
>> 
>> 
>> 
>> On 8/15/12 10:33 PM, "Qin Wu" <[e-mail]> wrote:
>> 
>>> Hi, Al:
>>> Thank for your general comments, I take some time to look at the
>>> standards for calculation again.
>>> please see my replies inline.
>>> 
>>> Regards!
>>> -Qin
>>> ----- Original Message -----
>>> From: "Al Morton" <[e-mail]>
>>> To: <[e-mail]>
>>> Sent: Thursday, August 02, 2012 5:01 AM
>>> Subject: [xrblock] comment on draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>> 
>>> 
>>>> Regarding:
>>>> http://tools.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-02.txt
>>>> 
>>>>>    Calculation Algorithm (CALg):3 bits
>>>>> 
>>>>>          000 - ITU-T P.564 Compliant Algorithm [P.564] (Voice)
>>>>>          001 - G.107 [G.107] (Voice)
>>>>>          010 - ETSI TS 101 329-5 Annex E [ ETSI] (Voice)
>>>>>          011 - ITU-T P.NAMS [P.NAMS] (Multimedia)
>>>>>          100 - ITU-T P.NBAMS [P.NBAMS] (Multimedia)
>>>>>          101~111 - Reserved for future extension.
>>>>> 
>>>>>       G.107 and P.564 and ETSI TS101 329-5 specify three
> Calculation
>>>>>       algorithms or MoS algorithms that are used to estimate
> speech
>>>>>       quality or conversation quality.  P.NAMS and P.NBAMS specify
>> two
>>>>>       MoS algorithms that are used to estimate multimedia quality
>>>>>       including video quality, audio quality and audio-video
>> quality.
>>>> 
>>>> Specifying the standard for the calculation is a good start, but
>>>> these standards all have options and input parameters that must
> also
>>>> be specified in order to know what the MOS means.
>>> 
>>>> For example, we need to know what codec was used, because G.726
>>>> carries an equipment Impairment factor that limits the upper bound
> on
>>>> MOS, while G.711 does not.
>>> 
>>> [Qin]: I agree MoS value depends on the codec that is in use. That's
>>> why, in the current draft, We use 7-bit payload Type in the metric
>>> block to signal what codec or payload format is  in use for
> reporting
>>> interval.
>>> Please see the definition of Payload Type (PT) in the section 3.2.1.
>>> 
>>>> 
>>>> Some of the input parameters will be assumed (e.g., at default
>>>> values) while others will be measured, and the distinction between
>>>> measured and assumed parameters should also be made.
>>> 
>>> [Qin]:
>>> The input parameters may come from Delay metric block, Discard
> metric
>>> block, Burst Gap metrics block, JB metrics block, loss concealment
>>> metric block, concelament seconds metrics block, these metrics
> blocks
>>> are defined in other XR Block drafts by XRBLOCK WG .
>>> 
>>> In QoE metric block, 4 or 5 pameters are defined, i.e., a.Segment
> Type
>>> b.MoS Type c.MoS algorithm d. MoS value In these parameters, a~c can
>>> be counted as default values while d is counted as measured value.
>>> The other default values rely on specific computation model defined
> in
>>> each standard, e.g., P.564, G.104, P.NAMS.
>>> Assume each standard only provide one algorithm or one model, we can
>>> use  the standards name and payload type(i.e.,codec) to identify
> each
>>> MoS algorithm used or each MoS model used.
>>> 
>>>> As I mentioned, metrics with many optional parameters are somewhat
>>>> difficult to identify in a simple way, as we found with the IPPM
>>>> Metrics Registry (which we withdrew when we found it was
> insufficient
>>>> to describe the measurement results in an exact way).
>>>> 
>>>> Colin Perkins suggested the possibility a registry for the
>> calculation alg.
>>>> A registry could provide a single value index to a complicated set
> of
>>>> input parameters and other assumptions, and this might work for
> you.
>>> 
>>> [Qin]: Yes,depends on the input parameters and option we got, we can
>>> calculate the different MoS values however the computation model is
>>> same and will not be affected by the input parameters and options.
>>> So maybe rather than registering MoS algorithm, we should register
>>> assessment model or computation model.
>>> We can use computation model defined in the standard and payload
> type
>>> as single value index.
>>> 
>>>> hope this helps,
>>>> Al
>>>> 
>>>> 
>>>> _______________________________________________
>>>> xrblock mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/xrblock
>>>> 
>>> _______________________________________________
>>> xrblock mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/xrblock
>> 
>> 
>> 
>> _______________________________________________
>> xrblock mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/xrblock