Re: [xrblock] Adam Roach's No Objection on draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: (with COMMENT)

[Ben Campbell <[e-mail]>]

Hi,

It looks like the discussion on Adam’s comments have reached resolution, and I think that is also mostly true for the other IESG comments. Authors; Please submit a new revision when you are ready.

Thanks!

Ben.

> On May 21, 2018, at 10:20 PM, Adam Roach <[e-mail]> wrote:
> 
> Thanks! Your proposed resolutions look good to me.
> 
> /a
> 
> On 5/21/18 10:18 PM, Huangyihong (Rachel) wrote:
>> Hi Adam,
>> 
>> Thanks for the comments. Please see inline.
>> 
>> BR,
>> Rachel
>> 
>>> -----Original Message-----
>>> From: Adam Roach [mailto:[e-mail]]
>>> Sent: Tuesday, May 22, 2018 8:56 AM
>>> To: The IESG
>>> Cc: [e-mail]; Shida Schubert;
>>> [e-mail]; [e-mail]; [e-mail]
>>> Subject: Adam Roach's No Objection on
>>> draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: (with COMMENT)
>>> 
>>> Adam Roach has entered the following ballot position for
>>> draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: No Objection
>>> 
>>> When responding, please keep the subject line intact and reply to all
>>> email addresses included in the To and CC lines. (Feel free to cut this
>>> introductory paragraph, however.)
>>> 
>>> 
>>> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
>>> for more information about IESG DISCUSS and COMMENT positions.
>>> 
>>> 
>>> The document, along with other ballot positions, can be found here:
>>> https://datatracker.ietf.org/doc/draft-ietf-xrblock-rtcweb-rtcp-xr-metrics/
>>> 
>>> 
>>> 
>>> ----------------------------------------------------------------------
>>> COMMENT:
>>> ----------------------------------------------------------------------
>>> 
>>> Thanks to the authors and contributors on this document. I have two
>>> substantive
>>> comments, and a handful of minor nits.
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §5.3.1:
>>> 
>>>>  application can use these metrics to calculate the Mean Opinion Score
>>>>  (MoS) values or Media Delivery Index (MDI) for their services.
>>> "...calculate Estimated Mean Opinion Score (eMOS) values..."
>>> 
>>> (MOS scores necessarily require human subjective evaluation of audio under
>>> carefully controlled circumstances -- any computer-generated metrics are
>>> inherently estimations, and should not be called MOS scores.)
>>> 
>> [Rachel]: You're right. Will change to "estimated Mean Opinion Score (MOS)" in the next version.
>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §5.3.1:
>>> 
>>>>  Error-resilience mechanisms, like RTP retransmission or FEC, are
>>>>  optional in RTCWEB because the overhead of the repair bits adding to
>>>>  the original streams.
>>> This is a little misleading. draft-ietf-rtcweb-rtp-usage says:
>>> 
>>>    WebRTC endpoints MUST follow the recommendations for FEC use given in
>>>    [I-D.ietf-rtcweb-fec].
>>> 
>>> And, in turn, draft-ietf-rtcweb-fec says:
>>> 
>>>    To support the functionality recommended above, implementations MUST
>>>    be able to receive and make use of the relevant FEC formats for their
>>>    supported audio codecs, and MUST indicate this support, as described
>>>    in Section 4.  Use of these formats when sending, as mentioned above,
>>>    is RECOMMENDED.
>>> 
>>> Rather than trying to reiterate this somewhat nuanced situation in *this*
>>> document, I recommend removing the entire sentence and fixing the rest of
>>> the
>>> paragraph. If you choose to keep it, please make sure it more clearly explains
>>> the level of FEC support required of RTCWEB implementations.
>>> 
>> [Rachel]: Your concern makes sense to me. I'll delete it as following:
>> 
>> OLD
>> 
>> "
>>    Error-resilience mechanisms, like RTP retransmission or FEC, are
>>    optional in RTCWEB because the overhead of the repair bits adding to
>>    the original streams.  But they do help to greatly reduce the impact
>>    of packet loss and enhance the quality of transmission.  Web
>>    applications could support certain repair mechanism after negotiation
>>    between both sides of browsers when needed.
>> "
>> 
>> NEW
>> "
>> Web applications could support certain RTP error-resilience mechanisms following the recommendations specified in [draft-ietf-rtcweb-rtp-usage].
>> "
>> 
>>> ================================================================
>>> ===========
>>> 
>>> The remainder of my comments are grammatical or editorial nits that should
>>> be
>>> fixed before progressing the document.
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §3:
>>> 
>>>>  The RTCP Sender Reports (SRs) and Receiver Reports (RRs) [RFC3550]
>>>>  exposes the basic metrics...
>>> "expose"
>>> 
>>>>  However, these metrics provides...
>>> "provide"
>>> 
>>> 
>>>>  For example, it may be useful to distinguish between
>>>>  packets lost and packets discarded due to late arrival, even though
>>>>  they have the same impact on the multimedia quality, it helps in
>>>>  identifying and diagnosing issues.
>>> This is a run-on sentence. Suggest: "...due to late arrival. Even though..."
>>> 
>>>>  The WebRTC application extracts the statistic from the browser by
>>>>  querying the getStats() API [W3C.WD-webrtc-20161124], but the browser
>>>>  currently only reports the local variables i.e., the statistics
>>>>  related to the outgoing RTP media streams and the incoming RTP media
>>>>  streams.
>>> I don't think the "currently" in this sentence is actually true any longer; and
>>> I expect it to get increasingly less true as time goes on. Consider rephrasing.
>>> 
>>>>  [I-D.ietf-rtcweb-rtp-usage] does not mandate the use of
>>>>  any RTCP XRs and since their usage is optional.
>>> This is not a sentence. Maybe remove "since"?
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §4:
>>> 
>>>>  For example, an application may choose
>>>>  to poll the stack for statistics every 1 second, in this case the
>>>>  underlying stack local will return the current snapshot of the local
>>>>  statistics (for incoming and outgoing media streams).
>>> This is a comma splice. Consider: "...every second. In this case..."
>>> 
>>>>  However it may
>>>>  return the same remote statistics as before for the remote
>>>>  statistics
>>> Add a comma after "However".
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §5:
>>> 
>>>>  Since following metrics are all defined in RTCP XR which is not
>>>>  mandated in WebRTC, all of them are local.
>>> "Since the following..."
>>>        ^^^
>>> 
>>> "...RTCP XR, which is..."
>>>            ^
>>> 
>>>>  However, if RTCP XR is
>>>>  supported by negotiation between two browsers, following metrics can
>>>>  also be generated remotely and be sent to local by RTCP XR packets.
>>> "...two browsers, the following..."
>>>                   ^^^
>>> 
>>>>  Following metrics are classified into 3 categories: network impact
>>> "The following metrics..."
>>>  ^^^
>>> 
>>> 
>>>>  viewpoint of application, e.g., bit rate, frames rate or jitter
>>> "...frame rate..."
>>> 
>>>>  WebRTC
>>>>  application can use these metrics to calculate the Mean Opinion Score
>>> "applications"
>>> 
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §5.1.1:
>>> 
>>>>  lost and duplicated packets.  Lost packets counts are useful for
>>> "Lost packet counts..."
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §5.1.3:
>>> 
>>>>  the two run length vectors to identify congestion-related losses,
>>>>  i.e., a router queue became overloaded causing delays and then
>>> Replace "i.e." ("that is") with "e.g." ("for example").
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §5.2.1:
>>> 
>>>>  The metric reports the cumulative size of the packets discarded in
>>>>  the interval, it is complementary to number of discarded packets.
>>> "...the interval. It is complementary..."
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §5.2.2:
>>> 
>>>>  For
>>>>  audio streams, a single RTP packet may contain one or multiple audio
>>>>  frames, each of which has a fixed length.
>>> I don't think this is generally true. Even in the case of WebRTC, I believe the
>>> default mode of operation for Opus is VBR, which results in frame sizes that
>>> vary from frame to frame.
>>> 
>>>>  The metrics in this category includes: number of discarded key
>>> "...category include:..."
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> 
>>> §5.3.1:
>>> 
>>>>  The un-repaired packets count and repaired loss count defined in
>>> "...unrepaired packet count..."
>>> 
>>>>  packets to lost packets.  Including this kind of metrics helps the
>>> Choose either "...these kinds of metrics..." or "...this kind of metric..."
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §5.3.2:
>>> 
>>>>  related losses, i.e., a router queue became overloaded causing delays
>>> Replace "i.e." with "e.g."
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §6:
>>> 
>>>>  identifiers relevant to RTP media in WebRTC.  These group of
>>>>  identifiers are defined on a ReportGroup corresponding to an
>>>>  Synchronization source (SSRC).  In practice the application need to
>>> "This group of identifiers..."
>>>  ^^^^
>>> 
>>> "...corresponding to a synchronization source..."
>>>                      ^
>>> 
>>> "...the application needs to..."
>>>                     ^^^^^
>>> 
>>>>  For XR metrics, it depends on
>>>>  two factors: 1) if it measured at the endpoint, 2) if it reported by
>>>>  the endpoint in an XR report.
>>> "...if it is measured..."
>>>           ^^
>>> 
>>> "...if it is reported..."
>>>           ^^
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §11.2:
>>> 
>>>>  [W3C.WD-webrtc-20161124]
>>>>             Sporny, M. and D. Longley, "WebRTC 1.0: Real-time
>>>>             Communication Between Browsers", World Wide Web
>>> Consortium
>>>>             WD WD-webrtc-20161124, November 2016,
>>>>             <https://www.w3.org/TR/2016/WD-webrtc-20161124>.
>>> The authorship of this document is incorrect (neither Sporny nor Longley are
>>> authors on WebRTC).
>>> 
>>> The URL points to an outdated version of the specification. Please update to
>>> https://www.w3.org/TR/2017/CR-webrtc-20171102/
>>> 
>>> 
>>>>  [W3C.WD-webrtc-stats-20161214]
>>>>             Alvestrand, H. and V. Singh, "Identifiers for
>>>>             WebRTC&#039;s Statistics API", World Wide Web Consortium
>>>>             WD WD-webrtc-stats-20161214, December 2016,
>>>>             <https://www.w3.org/TR/2016/WD-webrtc-stats-20161214>.
>>> Please replace "&#039;" with an apostrophe.
>>> 
>>> Please update the URL to
>>> https://www.w3.org/TR/2018/WD-webrtc-stats-20180519/
>>> 
>> [Rachel]: They'll be fixed in the next version. Thanks.
> 
>