Re: [xrblock] [pm-dir] request for an RFC 6390 review of http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt

["MORTON JR., ALFRED C (AL)" <[e-mail]>]

It appears this message to pm-dir (June 12) has not yet reached the XRBLOCK list,
but some similar comments have been made. I pass it along now:

==================================================================================

Dan,

I circulated your request for comments beyond pm-dir, among folks whose work is
referenced and reported in this draft. Several people commented,
and there was an independent request for the change to MOS already
suggested. I have consolidated the three additional comments below.

Please consider these comments during WGLC.

regards,
Al

-=-=-=-=-=-

Since it has just been approved in March 2013, [P.NBAMS-HR] can now be replaced by [P.1202.2].

The list of models referred to in this future RFC should be extended to include G.107.1, 
P.862.1, P.862.2 and P.863.

Important Point concerning MOS scales: 
There is no place in this draft RFC to define the type of MOS scale reported 
according to the terminology defined in P.800.1. 
Here is a simple example. Depending on whether you use or not a measurement result 
for end-to-end delay for the MOS computation (if not, you will use a 0 ms default value), 
the E-model will give you a MOS-CQE or a MOS-LQE (this is terminology from P.800.1). 
Another example is whether you use a Narrow-Band (NB) scale or a mixed Wide Band-NB scale. 
These are only audio examples and there may be similar examples for video, 
but apparently this draft applies to audio MOS alone. In other words, it may not be sufficient
to simply cite the Recommendation used for calculations, there are further details needed
to interpret the MOS value.

-=-=-=-=-=-

> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On Behalf
> Of MORTON JR., ALFRED C (AL)
> Sent: Tuesday, June 04, 2013 9:23 AM
> To: Romascanu, Dan (Dan); [e-mail]
> Cc: Benoit Claise (bclaise); Gonzalo Camarillo
> Subject: Re: [pm-dir] request for an RFC 6390 review of
> http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt
> 
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
> 
> Dan and pm-dir,
> 
> Consider your request made. Do we have a volunteer?
> 
> Here's a comment based on my scan of the draft:
> 
> Although the quality level is described as QoE,
> only estimates of MOS appear in the report blocks.
> So rather than relying on the industry's buzzword to
> describe the new block, it seems more accurate to simply say:
> 
> RTP Control Protocol (RTCP) Extended Report (XR) Blocks for MOS Metric
>                                Reporting
> 
> and the related global changes in the text.
> 
> And although there is continued discussion and evolution of the definition
> of QoE,
> MOS is just one aspect of QoE - the aspect we have understood for decades.
> 
> regards,
> Al

> -----Original Message-----
> From: Qin Wu [mailto:[e-mail]]
> Sent: Thursday, June 13, 2013 9:01 PM
> To: Alan Clark; Yaakov Stein; [e-mail]; Dan (Dan); MORTON
> JR., ALFRED C (AL); [e-mail]; [e-mail]
> Cc: [e-mail]
> Subject: RE: [xrblock] [pm-dir] request for an RFC 6390 review of
> http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt
> 
> Hi,all:
> It looks to me we are talking about two things.
> One is mos type,e.g., listening MoS, Conversational MoS, objective MoS,
> Subjective MoS.
> 
> The other is Mos reference or MoS Scale, e.g., audio with different
> bandwidth, video with different resolution.
> 
> Regarding mos type, I am wondering whether it can be inferred from payload
> type carried in the QoE XR Block,
> If the answer is no, I think we can add one new SDP attribute called "mos
> type", mapping "mos type" in the SDP
> to CAID used in the packet. Otherwise, we can infer mos type from payload
> type.
> 
> Regarding mos reference or MoS Scale, currently we have already supported
> 3options using "mosref"attribute,
> i.e.,lower resolution, higher resolution and combined lower and higher
> resolution.
> However we may have more than 2 scales,e.g., as Alan pointed out with
> speech/audio there are at least six
> different audio bandwidths/ sample rates used, in order to address this,
> one option is to define more than two option like lower resolution, middle
> resolution and high resolution and qualify them as typical resolution.
> Another option is to remove this 'mosref' and depend on other means to
> know these information.
> 
> Regards!
> -Qin
> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On Behalf
> Of Alan Clark
> Sent: Thursday, June 13, 2013 9:26 PM
> To: Yaakov Stein; [e-mail]; Dan (Dan); Al Morton; pm-
> [e-mail]; [e-mail]
> Cc: [e-mail]
> Subject: Re: [xrblock] [pm-dir] request for an RFC 6390 review of
> http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt
> 
> I agree that "MOS" is widely used and widely misunderstood. There are many
> different MOS ranges 1-5, 1-10, 1-11,....... and a plethora of other QoE
> related metrics
> 
> ITU P.800.1 helps to an extent for speech MOS - using MOS-[L|C]Q[E|O|S]
> where L = Listening, C = Conversational, E = estimated, O = objective and
> S
> = subjective.
> 
> Things are getting more complex - with speech/audio there are at least six
> different audio bandwidths/ sample rates used, in some cases the MOS/QoE
> score is mapped onto different ranges and sometimes onto the same range.
> With video we are already seeing 4k resolutions as well as 1080p, 1080i,
> 720p, 480i......
> 
> Yaakov also points out indirectly that there are many organizations that
> develop algorithms and standards related to the user perceived quality of
> applications (in the broad sense). ITU-T has a long history with speech
> MOS
> and a shorter history with video MOS measurement, ITU-R with video and
> audio
> MOS, and other organizations with data/ application QoE models.
> 
> Just a minor point on Yaakov's comments - P.563 obtained good correlation
> when averaged over a large set of conditions and speech samples (which is
> how ITU tested the algorithm) however performs poorly for individual
> calls,
> with MOS errors of up to +/- 40%.
> 
> Regards
> 
> Alan
> 
> 
> 
> On 6/13/13 8:45 AM, "Yaakov Stein" <[e-mail]> wrote:
> 
> > One should distinguish subjective QoE measures (averaging of
> measurements on
> > human subjects, e.g., MOS, MUSHRA, BT.500, Apdex)
> > and objective QoE (i.e., predictions of algorithms that have shown to
> > correlate well with subjective QoE, e.g., PESQ, PEAQ, P.563, PEVQ).
> >
> > Also note that there are several scales used for QoE type measures.
> > MOS is from 1 to 5 (and CCR MOS is from -3 to 3). Apdex is from 0 to 1.
> > R-factor is from 0 to 100.
> > So when retrieving a QoE measure one needs to know what to expect.
> >
> > Y(J)S
> >
> >
> > -----Original Message-----
> > From: [e-mail] [mailto:[e-mail]] On Behalf
> Of
> > [e-mail]
> > Sent: 13 June, 2013 09:48
> > To: [e-mail]; [e-mail]; [e-mail];
> [e-mail]
> > Cc: [e-mail]
> > Subject: Re: [pm-dir] [xrblock] request for an RFC 6390 review of
> > http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt
> >
> > Dear all,
> >
> > Quality of Experience (QoE), is a subjective measure of a customer's
> > experiences with a service. QoE systems will try to measure metrics that
> > customer will directly perceive as a quality parameter.
> > In my opinion QoE is a good expression to convey our intention.
> >
> > Regards
> >
> > Roland
> >
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: [e-mail] [mailto:[e-mail]] Im
> Auftrag von
> > Romascanu, Dan (Dan)
> > Gesendet: Dienstag, 4. Juni 2013 15:28
> > An: MORTON JR., ALFRED C (AL); [e-mail]; 'xrblock'
> > Cc: Benoit Claise (bclaise)
> > Betreff: Re: [xrblock] [pm-dir] request for an RFC 6390 review of
> > http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt
> >
> > (forwarding also to the xrblock list)
> >
> > This seems to me like a good comment - Al's proposal makes room in the
> future
> > for metrics different than the ones reporting MOS values to be used to
> > characterize QoE. I would like to hear the opinion of the authors, and
> other
> > participants in the WG.
> >
> > Thanks and Regards,
> >
> > Dan
> >
> >
> >> -----Original Message-----
> >> From: MORTON JR., ALFRED C (AL) [mailto:[e-mail]]
> >> Sent: Tuesday, June 04, 2013 4:23 PM
> >> To: Romascanu, Dan (Dan); [e-mail]
> >> Cc: Benoit Claise (bclaise); Gonzalo Camarillo
> >> Subject: RE: [pm-dir] request for an RFC 6390 review of
> >> http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt
> >>
> >> Dan and pm-dir,
> >>
> >> Consider your request made. Do we have a volunteer?
> >>
> >> Here's a comment based on my scan of the draft:
> >>
> >> Although the quality level is described as QoE, only estimates of MOS
> >> appear in the report blocks.
> >> So rather than relying on the industry's buzzword to describe the new
> >> block, it seems more accurate to simply say:
> >>
> >> RTP Control Protocol (RTCP) Extended Report (XR) Blocks for MOS Metric
> >>                                Reporting
> >>
> >> and the related global changes in the text.
> >>
> >> And although there is continued discussion and evolution of the
> >> definition of QoE, MOS is just one aspect of QoE - the aspect we have
> >> understood for decades.
> >>
> >> regards,
> >> Al
> >>
> >>
> >>> -----Original Message-----
> >>> From: [e-mail] [mailto:[e-mail]] On
> >>> Behalf Of Romascanu, Dan (Dan)
> >>> Sent: Monday, June 03, 2013 9:49 AM
> >>> To: [e-mail]
> >>> Cc: Benoit Claise (bclaise); Gonzalo Camarillo
> >>> Subject: [pm-dir] request for an RFC 6390 review of
> >>> http://www.ietf.org/id/draft-ietf-xrblock-rtcp-xr-qoe-08.txt
> >>>
> >>>
> >>> Hi PMDIR,
> >>>
> >>> As a co-chair of XRBLOCK WG, I would like to request the an RFC 6390
> >>> review of draft-ietf-xrblock-rtcp-xr-qoe-08. This document is in
> >>> WGLC until 6/17.
> >>>
> >>> Thanks and Regards,
> >>>
> >>> Dan
> >>>
> >>>
> >>> _______________________________________________
> >>> pm-dir mailing list
> >>> [e-mail]
> >>> https://www.ietf.org/mailman/listinfo/pm-dir
> > _______________________________________________
> > xrblock mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/xrblock
> > _______________________________________________
> > pm-dir mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/pm-dir
> > _______________________________________________
> > pm-dir mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/pm-dir
> 
> 
> _______________________________________________
> xrblock mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/xrblock