Re: [xrblock] Adam Roach's No Objection on draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: (with COMMENT)

"Huangyihong (Rachel)" <rachel.huang@huawei.com> Fri, 25 May 2018 00:39 UTC

Return-Path: <rachel.huang@huawei.com>
X-Original-To: xrblock@ietfa.amsl.com
Delivered-To: xrblock@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A173512EB49; Thu, 24 May 2018 17:39:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.201
X-Spam-Level:
X-Spam-Status: No, score=-4.201 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RhAAeUyWz-hO; Thu, 24 May 2018 17:39:11 -0700 (PDT)
Received: from huawei.com (lhrrgout.huawei.com [194.213.3.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A03B012EAA4; Thu, 24 May 2018 17:39:08 -0700 (PDT)
Received: from LHREML710-CAH.china.huawei.com (unknown [172.18.7.107]) by Forcepoint Email with ESMTP id 680D28D61A9A0; Fri, 25 May 2018 01:39:05 +0100 (IST)
Received: from NKGEML411-HUB.china.huawei.com (10.98.56.70) by LHREML710-CAH.china.huawei.com (10.201.108.33) with Microsoft SMTP Server (TLS) id 14.3.382.0; Fri, 25 May 2018 01:39:05 +0100
Received: from NKGEML513-MBX.china.huawei.com ([169.254.1.192]) by nkgeml411-hub.china.huawei.com ([10.98.56.70]) with mapi id 14.03.0382.000; Fri, 25 May 2018 08:38:57 +0800
From: "Huangyihong (Rachel)" <rachel.huang@huawei.com>
To: Ben Campbell <ben@nostrum.com>, "draft-ietf-xrblock-rtcweb-rtcp-xr-metrics@ietf.org" <draft-ietf-xrblock-rtcweb-rtcp-xr-metrics@ietf.org>
CC: The IESG <iesg@ietf.org>, "xrblock-chairs@ietf.org" <xrblock-chairs@ietf.org>, "shida.at.ietf@gmail.com" <shida.at.ietf@gmail.com>, "xrblock@ietf.org" <xrblock@ietf.org>
Thread-Topic: Adam Roach's No Objection on draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: (with COMMENT)
Thread-Index: AQHT8WelxxI9TTx7gUCoz7YxQT8WVqQ7D5kg//9/K4CABDKPgIAA3MNA
Date: Fri, 25 May 2018 00:38:57 +0000
Message-ID: <51E6A56BD6A85142B9D172C87FC3ABBB9C702661@nkgeml513-mbx.china.huawei.com>
References: <152695054904.8087.13733440385951037691.idtracker@ietfa.amsl.com> <51E6A56BD6A85142B9D172C87FC3ABBB9C6F78F5@nkgeml513-mbx.china.huawei.com> <dd5faa97-7448-2b49-5505-2c60cf460d68@nostrum.com> <A60D1D8C-76C6-4C29-9259-8340EC1CB541@nostrum.com>
In-Reply-To: <A60D1D8C-76C6-4C29-9259-8340EC1CB541@nostrum.com>
Accept-Language: zh-CN, en-US
Content-Language: zh-CN
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [10.134.153.152]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-CFilter-Loop: Reflected
Archived-At: <https://mailarchive.ietf.org/arch/msg/xrblock/G4GWuOknAIsOltbXZAxJNAnnkaw>
Subject: Re: [xrblock] Adam Roach's No Objection on draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: (with COMMENT)
X-BeenThere: xrblock@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Metric Blocks for use with RTCP's Extended Report Framework working group discussion list <xrblock.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/xrblock>, <mailto:xrblock-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/xrblock/>
List-Post: <mailto:xrblock@ietf.org>
List-Help: <mailto:xrblock-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/xrblock>, <mailto:xrblock-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 25 May 2018 00:39:23 -0000

Will do it today.

BR,
Rachel

> -----Original Message-----
> From: Ben Campbell [mailto:ben@nostrum.com]
> Sent: Friday, May 25, 2018 3:27 AM
> To: draft-ietf-xrblock-rtcweb-rtcp-xr-metrics@ietf.org; Huangyihong (Rachel)
> <rachel.huang@huawei.com>
> Cc: The IESG <iesg@ietf.org>rg>; xrblock-chairs@ietf.org;
> shida.at.ietf@gmail.com; xrblock@ietf.org
> Subject: Re: Adam Roach's No Objection on
> draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: (with COMMENT)
> 
> Hi,
> 
> It looks like the discussion on Adam’s comments have reached resolution, and
> I think that is also mostly true for the other IESG comments. Authors; Please
> submit a new revision when you are ready.
> 
> Thanks!
> 
> Ben.
> 
> > On May 21, 2018, at 10:20 PM, Adam Roach <adam@nostrum.com> wrote:
> >
> > Thanks! Your proposed resolutions look good to me.
> >
> > /a
> >
> > On 5/21/18 10:18 PM, Huangyihong (Rachel) wrote:
> >> Hi Adam,
> >>
> >> Thanks for the comments. Please see inline.
> >>
> >> BR,
> >> Rachel
> >>
> >>> -----Original Message-----
> >>> From: Adam Roach [mailto:adam@nostrum.com]
> >>> Sent: Tuesday, May 22, 2018 8:56 AM
> >>> To: The IESG
> >>> Cc: draft-ietf-xrblock-rtcweb-rtcp-xr-metrics@ietf.org; Shida
> >>> Schubert; xrblock-chairs@ietf.org; shida.at.ietf@gmail.com;
> >>> xrblock@ietf.org
> >>> Subject: Adam Roach's No Objection on
> >>> draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: (with COMMENT)
> >>>
> >>> Adam Roach has entered the following ballot position for
> >>> draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: No Objection
> >>>
> >>> When responding, please keep the subject line intact and reply to
> >>> all email addresses included in the To and CC lines. (Feel free to
> >>> cut this introductory paragraph, however.)
> >>>
> >>>
> >>> Please refer to
> >>> https://www.ietf.org/iesg/statement/discuss-criteria.html
> >>> for more information about IESG DISCUSS and COMMENT positions.
> >>>
> >>>
> >>> The document, along with other ballot positions, can be found here:
> >>> https://datatracker.ietf.org/doc/draft-ietf-xrblock-rtcweb-rtcp-xr-m
> >>> etrics/
> >>>
> >>>
> >>>
> >>> --------------------------------------------------------------------
> >>> --
> >>> COMMENT:
> >>> --------------------------------------------------------------------
> >>> --
> >>>
> >>> Thanks to the authors and contributors on this document. I have two
> >>> substantive comments, and a handful of minor nits.
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §5.3.1:
> >>>
> >>>>  application can use these metrics to calculate the Mean Opinion
> >>>> Score
> >>>>  (MoS) values or Media Delivery Index (MDI) for their services.
> >>> "...calculate Estimated Mean Opinion Score (eMOS) values..."
> >>>
> >>> (MOS scores necessarily require human subjective evaluation of audio
> >>> under carefully controlled circumstances -- any computer-generated
> >>> metrics are inherently estimations, and should not be called MOS
> >>> scores.)
> >>>
> >> [Rachel]: You're right. Will change to "estimated Mean Opinion Score (MOS)"
> in the next version.
> >>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §5.3.1:
> >>>
> >>>>  Error-resilience mechanisms, like RTP retransmission or FEC, are
> >>>> optional in RTCWEB because the overhead of the repair bits adding
> >>>> to  the original streams.
> >>> This is a little misleading. draft-ietf-rtcweb-rtp-usage says:
> >>>
> >>>    WebRTC endpoints MUST follow the recommendations for FEC use
> given in
> >>>    [I-D.ietf-rtcweb-fec].
> >>>
> >>> And, in turn, draft-ietf-rtcweb-fec says:
> >>>
> >>>    To support the functionality recommended above, implementations
> MUST
> >>>    be able to receive and make use of the relevant FEC formats for their
> >>>    supported audio codecs, and MUST indicate this support, as described
> >>>    in Section 4.  Use of these formats when sending, as mentioned
> above,
> >>>    is RECOMMENDED.
> >>>
> >>> Rather than trying to reiterate this somewhat nuanced situation in
> >>> *this* document, I recommend removing the entire sentence and fixing
> >>> the rest of the paragraph. If you choose to keep it, please make
> >>> sure it more clearly explains the level of FEC support required of
> >>> RTCWEB implementations.
> >>>
> >> [Rachel]: Your concern makes sense to me. I'll delete it as following:
> >>
> >> OLD
> >>
> >> "
> >>    Error-resilience mechanisms, like RTP retransmission or FEC, are
> >>    optional in RTCWEB because the overhead of the repair bits adding to
> >>    the original streams.  But they do help to greatly reduce the impact
> >>    of packet loss and enhance the quality of transmission.  Web
> >>    applications could support certain repair mechanism after negotiation
> >>    between both sides of browsers when needed.
> >> "
> >>
> >> NEW
> >> "
> >> Web applications could support certain RTP error-resilience mechanisms
> following the recommendations specified in [draft-ietf-rtcweb-rtp-usage].
> >> "
> >>
> >>>
> ================================================================
> >>> ===========
> >>>
> >>> The remainder of my comments are grammatical or editorial nits that
> >>> should be fixed before progressing the document.
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §3:
> >>>
> >>>>  The RTCP Sender Reports (SRs) and Receiver Reports (RRs) [RFC3550]
> >>>> exposes the basic metrics...
> >>> "expose"
> >>>
> >>>>  However, these metrics provides...
> >>> "provide"
> >>>
> >>>
> >>>>  For example, it may be useful to distinguish between  packets lost
> >>>> and packets discarded due to late arrival, even though  they have
> >>>> the same impact on the multimedia quality, it helps in  identifying
> >>>> and diagnosing issues.
> >>> This is a run-on sentence. Suggest: "...due to late arrival. Even though..."
> >>>
> >>>>  The WebRTC application extracts the statistic from the browser by
> >>>> querying the getStats() API [W3C.WD-webrtc-20161124], but the
> >>>> browser  currently only reports the local variables i.e., the
> >>>> statistics  related to the outgoing RTP media streams and the
> >>>> incoming RTP media  streams.
> >>> I don't think the "currently" in this sentence is actually true any
> >>> longer; and I expect it to get increasingly less true as time goes on.
> Consider rephrasing.
> >>>
> >>>>  [I-D.ietf-rtcweb-rtp-usage] does not mandate the use of  any RTCP
> >>>> XRs and since their usage is optional.
> >>> This is not a sentence. Maybe remove "since"?
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §4:
> >>>
> >>>>  For example, an application may choose  to poll the stack for
> >>>> statistics every 1 second, in this case the  underlying stack local
> >>>> will return the current snapshot of the local  statistics (for
> >>>> incoming and outgoing media streams).
> >>> This is a comma splice. Consider: "...every second. In this case..."
> >>>
> >>>>  However it may
> >>>>  return the same remote statistics as before for the remote
> >>>> statistics
> >>> Add a comma after "However".
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §5:
> >>>
> >>>>  Since following metrics are all defined in RTCP XR which is not
> >>>> mandated in WebRTC, all of them are local.
> >>> "Since the following..."
> >>>        ^^^
> >>>
> >>> "...RTCP XR, which is..."
> >>>            ^
> >>>
> >>>>  However, if RTCP XR is
> >>>>  supported by negotiation between two browsers, following metrics
> >>>> can  also be generated remotely and be sent to local by RTCP XR
> packets.
> >>> "...two browsers, the following..."
> >>>                   ^^^
> >>>
> >>>>  Following metrics are classified into 3 categories: network impact
> >>> "The following metrics..."
> >>>  ^^^
> >>>
> >>>
> >>>>  viewpoint of application, e.g., bit rate, frames rate or jitter
> >>> "...frame rate..."
> >>>
> >>>>  WebRTC
> >>>>  application can use these metrics to calculate the Mean Opinion
> >>>> Score
> >>> "applications"
> >>>
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §5.1.1:
> >>>
> >>>>  lost and duplicated packets.  Lost packets counts are useful for
> >>> "Lost packet counts..."
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §5.1.3:
> >>>
> >>>>  the two run length vectors to identify congestion-related losses,
> >>>> i.e., a router queue became overloaded causing delays and then
> >>> Replace "i.e." ("that is") with "e.g." ("for example").
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §5.2.1:
> >>>
> >>>>  The metric reports the cumulative size of the packets discarded in
> >>>> the interval, it is complementary to number of discarded packets.
> >>> "...the interval. It is complementary..."
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §5.2.2:
> >>>
> >>>>  For
> >>>>  audio streams, a single RTP packet may contain one or multiple
> >>>> audio  frames, each of which has a fixed length.
> >>> I don't think this is generally true. Even in the case of WebRTC, I
> >>> believe the default mode of operation for Opus is VBR, which results
> >>> in frame sizes that vary from frame to frame.
> >>>
> >>>>  The metrics in this category includes: number of discarded key
> >>> "...category include:..."
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>>
> >>> §5.3.1:
> >>>
> >>>>  The un-repaired packets count and repaired loss count defined in
> >>> "...unrepaired packet count..."
> >>>
> >>>>  packets to lost packets.  Including this kind of metrics helps the
> >>> Choose either "...these kinds of metrics..." or "...this kind of metric..."
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §5.3.2:
> >>>
> >>>>  related losses, i.e., a router queue became overloaded causing
> >>>> delays
> >>> Replace "i.e." with "e.g."
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §6:
> >>>
> >>>>  identifiers relevant to RTP media in WebRTC.  These group of
> >>>> identifiers are defined on a ReportGroup corresponding to an
> >>>> Synchronization source (SSRC).  In practice the application need to
> >>> "This group of identifiers..."
> >>>  ^^^^
> >>>
> >>> "...corresponding to a synchronization source..."
> >>>                      ^
> >>>
> >>> "...the application needs to..."
> >>>                     ^^^^^
> >>>
> >>>>  For XR metrics, it depends on
> >>>>  two factors: 1) if it measured at the endpoint, 2) if it reported
> >>>> by  the endpoint in an XR report.
> >>> "...if it is measured..."
> >>>           ^^
> >>>
> >>> "...if it is reported..."
> >>>           ^^
> >>>
> >>> --------------------------------------------------------------------
> >>> -------
> >>>
> >>> §11.2:
> >>>
> >>>>  [W3C.WD-webrtc-20161124]
> >>>>             Sporny, M. and D. Longley, "WebRTC 1.0: Real-time
> >>>>             Communication Between Browsers", World Wide Web
> >>> Consortium
> >>>>             WD WD-webrtc-20161124, November 2016,
> >>>>             <https://www.w3.org/TR/2016/WD-webrtc-20161124>.
> >>> The authorship of this document is incorrect (neither Sporny nor
> >>> Longley are authors on WebRTC).
> >>>
> >>> The URL points to an outdated version of the specification. Please
> >>> update to https://www.w3.org/TR/2017/CR-webrtc-20171102/
> >>>
> >>>
> >>>>  [W3C.WD-webrtc-stats-20161214]
> >>>>             Alvestrand, H. and V. Singh, "Identifiers for
> >>>>             WebRTC&#039;s Statistics API", World Wide Web
> Consortium
> >>>>             WD WD-webrtc-stats-20161214, December 2016,
> >>>>
> <https://www.w3.org/TR/2016/WD-webrtc-stats-20161214>.
> >>> Please replace "&#039;" with an apostrophe.
> >>>
> >>> Please update the URL to
> >>> https://www.w3.org/TR/2018/WD-webrtc-stats-20180519/
> >>>
> >> [Rachel]: They'll be fixed in the next version. Thanks.
> >
> >