Re: [xrblock] Call for adopting xr-quality-monitoring-06 as WG item

["Charles Eckel (eckelcu)" <[e-mail]>]

(as an individual)

It seems the merge efforts resulted in a combination of arguably
separate metrics within a single draft. I think this highlights some of
the reasons why we had two drafts to start with. I agree with Colin's
objection per the principles established of the monarch draft. I think
it is worth looking at the various MOS types and seeing how their usage
is simplified if this draft is in fact separated into three component
drafts. At initial glance, there is some simplification. Furthermore,
seeing as the existing draft already contains the restriction that
two segment types cannot be present in the same metric block, splitting
into multiple drafts does not result in any additional duplication of
information within the resulting metric blocks.

Cheers,
Charles

> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On
Behalf Of Qin Wu
> Sent: Tuesday, December 20, 2011 7:24 PM
> To: Colin Perkins
> Cc: [e-mail]
> Subject: Re: [xrblock] Call for adopting xr-quality-monitoring-06 as
WG item
> 
> Hi, Colin and Chairs:
> >----- Original Message -----
> >From: "Colin Perkins" <[e-mail]>
> >To: "Qin Wu" <[e-mail]>
> >Cc: "Shida Schubert" <[e-mail]>; <[e-mail]>
> >Sent: Wednesday, December 21, 2011 1:06 AM
> >Subject: Re: [xrblock] Call for adopting xr-quality-monitoring-06 as
WG item
> 
> 
> >Qin,
> 
> >See inline.
> >Colin
> 
> 
> >>On 20 Dec 2011, at 01:54, Qin Wu wrote:
> >> Hi, Colin:
> >> I understand your concerns. It is painful, but that's the tradeoff
we should made.
> >> Please see my clarification belows.
> >> ----- Original Message -----
> >> From: "Colin Perkins" <[e-mail]>
> >> To: "Shida Schubert" <[e-mail]>
> >> Cc: <[e-mail]>
> >> Sent: Tuesday, December 20, 2011 1:13 AM
> >> Subject: Re: [xrblock] Call for adopting xr-quality-monitoring-06
as WG item
> >>
> >>
> >>> Shida,
> >>>
> >>> I don't think this draft is ready to adopt:
> >>>
> >>> - The single-stream metric mixes audio and video MOS into a single
metric block, with a grab-bag
> of MOS type and calculation algorithm parameters. It seems that little
thought has gone into designing
> >>>the XR block format to prohibit invalid combinations (e.g., MT =
MOS-V with CALg = G.107), and
> there is no guidance on which algorithm is appropriate for which MOS
type.
> >>>
> >> [Qin]:  We do have guidance to prevent such combination  in three
places
> >> a. See MoS Type (MT) field definition, it said:
> > "
> >>      If MoS type is MoS-LQ and MoS-CQ, the MoS value can be
calculated
> >>      based on ITU-T G.107[G.107], ITU-T P.564 [P.564]or ETSI TS 101
> >>      329-5 [ETSI], if the Mos type is MoS-V or MoS-AV, the Mos
value
> >>      can be calculated based on ITU-T P.NAMS [P.NAMS]or ITU-T
P.NBAMS
> >>      [P.NBAMS].
> >> "
> >> b. Also see Mos Type (MT) field defintion, it said:
> >> "
> >>     If MoS Type does not match the MoS algorithm in the report
(e.g.,
> >>      specify a voice MOS algorithm for a video quality MOS), the
> >>      receiver should also discard this report.
> >> "
> >> c.  See Calculation Algorithm (CALg) field definition
> >> It distinguish Audio MoS algorithm from video MoS algorithm by
adding "(voice)" into Audio MoS
> algorithm.
> 
> >I agree that there is guidance to avoid invalid combinations.
However, a better design would prevent
> invalid combinations by appropriate choice of report block structure.
> 
> >I still see no guidance on which algorithm is appropriate for which
MOS type.
> 
> [Qin]: I get your point. In the current draft, we list 5 MoS
algorithms, 3 MOS algorithms specifed in
> the G.107 and P.564 and ETSI TS101 329-5 are only used to estimate
speech quality or conversation
> quality. 2 MoS algorithms specified in the P.NAMS and P.NBAMS can be
used to estimate either audio
> quality or video quality or multimedia quality. Also I like to propose
to add one MoS Type MoS-A into
> the draft to distinguish conversation MoS and speech MoS from audio
MoS.
> 
> If you think this clarificatin does address your comments, I can put
them into the update of the
> draft.
> 
> >>> The format seems to follow the RFC 3611 model, throwing together a
set of unrelated parameters,
> rather than the approach documents in the monarch draft.
> >>
> >> [Qin]: In this draft, we are not allowed to put "single stream
segment","multi-layer segment" and
> "multi-channel segment" into one XR Block Report. please see the note
in the segement i field
> definition
> >> it said:
> >> "
> >>      Note that in these
> >>      three segment type,any two segment types can not be present in
the
> >>      same metric block.
> >> "
> >> That is to say if we choose to use "single stream segment" to
report MoS, we will not use "multi-
> channel segment" or "multi-layer segment" . If we choose to use
multi-channel segment, we will not use
> >> "single stream segment" or "multi-layer segment".
> >>
> >> Also we may need to measure MoS-CQ using three different audio MoS
algorithms, e.g.,
> >> Using ITU-T P.564 to calculate MoS-CQ1
> >> Using G.107 to calculate MoS-CQ2
> >> Using ETSI TS 101 329-5 to calculate MoS-CQ3
> >>
> >> In this case,  MoS-CQ1 and MoS-CQ2 and MoS-CQ3 are all MoS value,
but using different audio MoS
> algorithm, so the question is how to report them, we have two options:
> >> 1. Report each MoS-CQ value carrried in the "single stream segment"
using three XR Blocks
> >> 2. Report three MoS-CQ vlaues carried in three "single stream
segment" using only one XR Block.
> >> I think the option 2 is a better approach, this approach only
report MoS value related to one
> single MoS Type and doesn't put a set of unrelated parameters together
and therefore doesn't conflict
> the rule >>we set in MONARCH.
> >> However if we also measure MoS-LQ using three different audio MoS
algorithms and report MoS-CQ
> value and MoS-LQ value carried in two "single stream segment" using
one single XR Block, that >>does
> mean "put a set of unrelated parameters together" and therefore
conflict with Monarch.
> >>
> >> So my suggestion is to add a sentence to say only one MoS type is
allowed to report in one single
> XR Block.
> >>
> >> If you agree with this, I can revise the draft to make this clear.
> 
> >This change doesn't address my concern, which is that a single draft
is describing three different
> types of report block. The monarch draft was intended to recommend
different report blocks be
> developed >in different drafts, so they can be implemented separately,
in a mix-and-match style,
> rather than being combined into a single draft.
> 
> [Qin]: I agree, I have no strong opinon on this. I am okay to either
put them toether in one single
> draft or document them separately.
> but my difficulty is these three type of report blocks all report MoS
value and are all for QoE metric
> reporting. I am not worried about block type space but I do worry
> about or need to figure out how to distinguish one another. But if we
put them together, we don't have
> such issue. Also it is not difficult for the user to place his order
by choosing different segment
> type
> rather than place his order by choosing different block type for the
same purpose(QoE metric
> reporting).
> 
> >>> - The multi-layer segment is ill-specified: does it support audio
only, video only, or both
> depending on the Media Type bit?
> >>
> >> [Qin]: multi-layer segment does support both video and audio,
> 
> >None of the field descriptions in Section 3.2.2 mention audio;
they're all video only.
> 
> [Qin]: You are right, I check RFC6190, draft-ietf-avt-rtp-mvc-01, it
looks to me these specification
> only support video. Audio seems to be carried in the different RTP
session or different
> RTP stream with different SSRC. And then using CNAME to group audio
with layered video. Am I right?
> If yes, can we say multi-layer segment only support video?
> 
> >>> Which calculation algorithm makes sense here, since there is no
guidance given, and it's possible
> to specify any algorithm?
> >>>
> >> [Qin]: currently we only allow 5 MoS algorithms.since three Audio
MoS algorithms have already been
> standardized and the other two are in standardization progress.
> 
> >This doest address my question. What algorithms make sense in the
context of layered media?
> 
> [Qin]: Do we really have layered audio using RTP payload format?
> If no, I think P.NAMS or PNBAMS is sufficient enough for layed media.
> 
> >>> What is the SSID field? How it is used?
> >>
> >> [Qin]:SSID is sub stream identifier and can be used to identifier
each layer if all the layers are
> carried in the same RTP stream.
> 
> >More detail is needed in the draft.
> 
> [Qin]: Okay.
> 
> >>> It's not sufficient to simply say that a "NAL unit type is one
example", the draft needs to give
> normative rules for the use of this field.
> >>
> >> [Qin]:Agree.
> >>
> >>> - The media type, MOS type, and calculation algorithm for the
multi-channel segment are ill-
> specified in similar ways to the multi-layer segment.
> >>
> >> [Qin]: Yes, you are right.
> >>
> >>> Also, it's not clear that the channel mapping in RFC 3550 Section
4.1 is the only one in use.
> >>
> >> [Qin]: Good point, but that's what we have in mind currently.
> >>
> >>> - In general, it's not clear to me that including multiple
different types of MOS report (single-
> stream, multi-layer, and multi-channel) in a single draft fits with
the monitoring architecture, which
> encourages >small, single-use, XR block definitions. I would suggest
that the three different MOS
> report types would be better defined as three drafts: there is little
in common between them.
> >>
> >> [Qin]: As I clarified above, these three MOS report types can not
be put into one single XR
> Block.They are excluded between one another.
> >> But If WG think it is a good idea to merge them together, I am okay
with that.
> 
> I agree that they should not be included in a single XR block. My
question, however, was whether they
> should be in a single draft.
> 
> [Qin]: See above, also I like to hear chairs' opinion.
> 
> >>> - The draft provides no meaningful guidelines for registration of
new MOS algorithms, and has an
> inadequate IANA registry.
> >>
> >> [Qin]: We do have in the (-06), section 5.4 of
xr-quality-monitoring-06
> 
> >Much more detail on the registration procedures, and guidelines for
what should be registered, is
> needed.
> 
> [Qin]: Okay.
> 
> >> "
> >> 5.4.  New registry of calculation algorithms
> >>
> >>   This document creates a new registry to be called "RTCP XR QoE
metric
> >>   block - Calculation Algorithm" as a sub-registry of the "RTP
Control
> >>   Protocol Extended Reports (RTCP XR) Block Type Registry".
Policies
> >>   for this new registry are as follows:
> >>
> >>   o  Entries in the registry are integers.  The valid range is 0 to
7
> >>      corresponding to the 3-bit field "CAlg" in the block.  Values
are
> >>      to be recorded in decimal.
> >>
> >>   o  Initial assignments are as follows:
> >>
> >>      1.  ITU-T P.564 Compliant Algorithm [P.564] (Voice)
> >>      2.  G.107 [G.107] (Voice)
> >>      3.  ETSI TS 101 329-5 Annex E ETSI] (Voice)
> >>      4.  ITU-T P.NAMS [P.NAMS]
> >>      5.  ITU-T P.NBAMS [P.NBAMS]
> >> "
> >>> Hope this makes sense.
> >>>
> >>> Colin
> >>>
> >>>
> >>> On 19 Dec 2011, at 11:55, Shida Schubert wrote:
> >>>> This is a call to see whether people are happy
> >>>> with the latest xr-quality-monitoring-06 draft for addressing
> >>>> the milestone "QoE Metrics Reporting".
> >>>>
> >>>>
http://tools.ietf.org/html/draft-wu-xrblock-rtcp-xr-quality-monitoring-0
6
> >>>>
> >>>> xr-que-00 which used to be an AVT draft also addresses
> >>>> the same milestone.
> >>>>
> >>>> http://tools.ietf.org/id/draft-clark-xrblock-rtcp-xr-qoe-00.txt
> >>>>
> >>>> The xr-quality-monitoring-06 claims to merge the two
> >>>> into one.
> >>>>
> >>>> Please response to express your option for the followings.
> >>>>
> >>>> 1. Adopt the draft-wu-xrblock-rtcp-xr-quality-monitoring-06 as a
WG item.
> >>>>
> >>>> 2. Not ready to adopt.
> >>>>  * Would be very helpful if you provide your reasonings.
> >>>>
> >>>> Regards
> >>>> Shida (as chair)
> >>> _______________________________________________
> >>> xrblock mailing list
> >>> [e-mail]
> >>> https://www.ietf.org/mailman/listinfo/xrblock
> >>
> >>
> >> --
> >> Colin Perkins
> >> http://csperkins.org/
> >>
> >>
> >>
> >> _______________________________________________
> >> xrblock mailing list
> >> [e-mail]
> >> https://www.ietf.org/mailman/listinfo/xrblock
> > _______________________________________________
> > xrblock mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/xrblock
> 
> 
> 
> --
> Colin Perkins
> http://csperkins.org/
> 
> 
> 
> _______________________________________________
> xrblock mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/xrblock