Re: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)
Roman Danyliw <rdd@cert.org> Fri, 21 June 2019 14:09 UTC
Return-Path: <rdd@cert.org>
X-Original-To: video-codec@ietfa.amsl.com
Delivered-To: video-codec@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 96FFE12026E; Fri, 21 Jun 2019 07:09:00 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=cert.org
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id G0qHWfS75_dJ; Fri, 21 Jun 2019 07:08:57 -0700 (PDT)
Received: from taper.sei.cmu.edu (taper.sei.cmu.edu [147.72.252.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5B468120114; Fri, 21 Jun 2019 07:08:57 -0700 (PDT)
Received: from korb.sei.cmu.edu (korb.sei.cmu.edu [10.64.21.30]) by taper.sei.cmu.edu (8.14.7/8.14.7) with ESMTP id x5LE8hXV020821; Fri, 21 Jun 2019 10:08:43 -0400
DKIM-Filter: OpenDKIM Filter v2.11.0 taper.sei.cmu.edu x5LE8hXV020821
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cert.org; s=yc2bmwvrj62m; t=1561126123; bh=qyEPNnMPGEeBuLlZ15gPNa/JCGJ6QbPqYRb6yfNY0zc=; h=From:To:CC:Subject:Date:References:In-Reply-To:From; b=dvSTN7TGnXvsAT5ownYABrrMtGG7GgOJ7Lg6a1Q5HkAdyYNITkmdqEW0OqDmcIfhm v9S7Q197L5rhXgFFzmQ/54jyWQGD6y3aQxjHbn2tE+WcQqllndeWUzxi02m+0oHk3q A+y20Y6Bs6iJV4Fsk/exmaEEbyUYLx8utgckZipE=
Received: from CASSINA.ad.sei.cmu.edu (cassina.ad.sei.cmu.edu [10.64.28.249]) by korb.sei.cmu.edu (8.14.7/8.14.7) with ESMTP id x5LE8dbi018908; Fri, 21 Jun 2019 10:08:39 -0400
Received: from MARATHON.ad.sei.cmu.edu ([10.64.28.250]) by CASSINA.ad.sei.cmu.edu ([10.64.28.249]) with mapi id 14.03.0439.000; Fri, 21 Jun 2019 10:08:38 -0400
From: Roman Danyliw <rdd@cert.org>
To: Filippov Alexey <Alexey.Filippov@huawei.com>, Adam Roach <adam@nostrum.com>, The IESG <iesg@ietf.org>
CC: "netvc-chairs@ietf.org" <netvc-chairs@ietf.org>, "video-codec@ietf.org" <video-codec@ietf.org>, "draft-ietf-netvc-requirements@ietf.org" <draft-ietf-netvc-requirements@ietf.org>, Mo Zanaty <mzanaty@cisco.com>
Thread-Topic: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)
Thread-Index: AQHVIL2m9AWiZ72osE6epr04IDyWcKaaGJ4AgAwKMSA=
Date: Fri, 21 Jun 2019 14:08:38 +0000
Message-ID: <359EC4B99E040048A7131E0F4E113AFC01B33A0CEA@marathon>
References: <156030268519.5895.7315446863069831893.idtracker@ietfa.amsl.com> <8820dcb9272f4de7b3ebb7cba68052f4@huawei.com>
In-Reply-To: <8820dcb9272f4de7b3ebb7cba68052f4@huawei.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [10.64.22.6]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Archived-At: <https://mailarchive.ietf.org/arch/msg/video-codec/JGFihk7dnyv4ln_dS88OZjL2Hn4>
Subject: Re: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)
X-BeenThere: video-codec@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Video codec BoF discussion list <video-codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/video-codec>, <mailto:video-codec-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/video-codec/>
List-Post: <mailto:video-codec@ietf.org>
List-Help: <mailto:video-codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/video-codec>, <mailto:video-codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 21 Jun 2019 14:09:01 -0000
Hi! > -----Original Message----- > From: iesg [mailto:iesg-bounces@ietf.org] On Behalf Of Filippov Alexey > Sent: Thursday, June 13, 2019 1:15 PM > To: Roman Danyliw <rdd@cert.org>; Adam Roach <adam@nostrum.com>; > The IESG <iesg@ietf.org> > Cc: netvc-chairs@ietf.org; video-codec@ietf.org; draft-ietf-netvc- > requirements@ietf.org; Mo Zanaty <mzanaty@cisco.com> > Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: > (with DISCUSS and COMMENT) > > Dear Roman, > > Thank you a lot for your comments and raising the important questions. > > > How should a set of QPs be specified? > As you probably know, a set of available QP values can be codec specific. > Usually, a QP set is selected to cover medium bit-rate range that is > considered to be the most complex for compressing. We proposed to set up > the assessment process not only for this range but also for low and high QP > ranges to cover a wider range of applications. According to my > understanding, concrete QP values should be specified for candidate codecs > in draft-ietf-netvc-testing See below. Can you help with a specific pointer into draft-ietf-netvc-testing which lists candidate codecs+QPs. > > > How should the quality values be calculated? > It is explained in Section 4.1, namely: “To assess the quality of output > (decoded) sequences, two indexes, PSNR [3] and MS-SSIM [3,11] are > separately computed. In the case of the YCbCr color format, PSNR should be > calculated for each color plane whereas MS-SSIM is calculated for luma > channel only. In the case of the RGB color format, both metrics are > computed for R, G and B channels. Thus, for each sequence, 30 RD-points for > PSNR (i.e. three RD-curves, one for each channel) and 10 RD-points for MS- > SSIM (i.e. one RD-curve, for luma channel only) should be calculated in the > case of YCbCr. If content is encoded as RGB, 60 RD-points (30 for PSNR and > 30 for MS-SSIM) should be calculated, i.e. three RD-curves (one for each > channel) are computed for PSNR as well as three RD-curves (one for each > channel) for MS-SSIM.” In references [3, 11], these 2 quality assessment > metrics and the ways of how to calculate them are described in detail. Got it. Thanks. > >-- What does the text “(in a separate document on Internet video codec > testing)” mean? > > Per “A list of video sequences that should be used for testing as well as the > 10 QP values for the reference codec are defined in a separate document ", > what document is that? Is it draft-ietf-netvc-testing? > According to my understanding, it is draft-ietf-netvc-testing. Can you please help me with the specific section references in draft-ietf-netvc-testing as I'm not seeing it -- where is the explicit reference codec being named and its associate 10 QP values? For what it's worth I see Section 4.1 of this draft saying that: As the reference for evaluation, state-of-the-art video codecs such as HEVC/H.265 [4,5] or VP9 must be used. The reference source code of the HEVC/H.265 codec can be found at [6]. The HEVC/H.265 codec must be configured according to [13] and Table 9. I'm looking for something that either unambiguously points to the reference codec+QPs; or clearer language that says that all of this is out of scope in this and/or the -testing draft, but is clear on what input into the processes of this draft is required. > >What does “codec implementation (for both an encoder and a decoder) > should cover the worst case of computational complexity, memory > bandwidth, and physical memory size” mean? > I’d like to thank Adam for his comment. I also think that his formulation > ("...should take into consideration the worst-case...") is clearer. > A more detailed explanation on this Section can be found in my response to > the secdir reviewer: https://mailarchive.ietf.org/arch/msg/secdir/QYx- > hl07aec5XhgqkldqgWFG3Is I'd like to offer a further refinement of what Adam proposed: Original: However, it is worth noting that a codec implementation (for both an encoder and a decoder) should cover the worst case of computational complexity, memory bandwidth, and physical memory size (e.g., for decoded pictures used as references). Adam: However, it is worth noting that a codec implementation (for both an encoder and a decoder) should take into consideration the worst-case computational complexity, memory bandwidth, and physical memory size (e.g., for decoded pictures used as references). Roman+Adam: However, it is worth noting that a codec implementation (for both an encoder and a decoder) should take into consideration the worst-case computational complexity, memory bandwidth, and physical memory size needed to processes the input (e.g., the decoded pictures used as references). > >Please add additional language that codec should be written in a defensive > style as they will be processing untrusted input. > Good point. Thank you. I’ll add it. Regards, Roman > > -- > Best regards, > Alexey Filippov > > -----Original Message----- > From: Roman Danyliw via Datatracker [mailto:noreply@ietf.org] > Sent: Wednesday, June 12, 2019 4:25 AM > To: The IESG <iesg@ietf.org> > Cc: draft-ietf-netvc-requirements@ietf.org; Mo Zanaty > <mzanaty@cisco.com>; netvc-chairs@ietf.org; mzanaty@cisco.com; video- > codec@ietf.org > Subject: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with > DISCUSS and COMMENT) > > Roman Danyliw has entered the following ballot position for > draft-ietf-netvc-requirements-09: Discuss > > When responding, please keep the subject line intact and reply to all email > addresses included in the To and CC lines. (Feel free to cut this introductory > paragraph, however.) > > > Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html > for more information about IESG DISCUSS and COMMENT positions. > > > The document, along with other ballot positions, can be found here: > https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/ > > > > ---------------------------------------------------------------------- > DISCUSS: > ---------------------------------------------------------------------- > > (1) I worry that the level of detail in the in the Compression Performance > Evaluation (Section 4.1) is insufficient for implementation. Specifically: > > (a) Per “Initially, for the codec selected as a reference one (e.g., HEVC or > VP9), a set of 10 QP quantization parameter) values should be specified (in a > separate document on Internet video codec testing) and corresponding > quality values should be calculated.” > > -- How should a set of QPs be specified? > > --How should the quality values be calculated? > > -- What does the text “(in a separate document on Internet video codec > testing)” mean? > > (b) Per “A list of video sequences that should be used for testing as well as > the 10 QP values for the reference codec are defined in a separate document > ", what document is that? Is it draft-ietf-netvc-testing? > > (2) Per the Security Considerations Section (Section 5) > > -- What does “codec implementation (for both an encoder and a decoder) > should cover the worst case of computational complexity, memory > bandwidth, and physical memory size” mean? > > -- Please add additional language that codec should be written in a > defensive style as they will be processing untrusted input. > > > ---------------------------------------------------------------------- > COMMENT: > ---------------------------------------------------------------------- > > (1) There is something odd about the document formatting – the title and > the first author last name in the footer is wrapped in “< … >” > > (2) It would be helpful to forward reference that acronyms are explained in > Appendix A. > > (3) This draft uses the words should and must to prescribe action. Why > wasn’t > RFC2119 cited to explain these words? > > (4) Section 2.0. A reference to explain “YCbCr 4:2:0” would be helpful > > (5) Section 2.1. Per “high encoder complexity” and “decoding complexity”, I > initiate read that as a qualitative measure. However, the text says “up to > 10x and more” so that implies some quantitative measure. What is that? > > (6) Section 2.1. Expand QP values on first use > > (7) Section 2.x. The language around content doesn’t appear to be consist. > For example: > > -- Section 2.1, Internet Video Streaming says “movies, TV-series and shows, > and animation.” > > -- Section 2.2, IPTV says “television content” > > -- Section 2.5, Screen casting says “business presentations …, animation > (cartoons), gaming content, data visualization, …, virtual desktop > infrastructure (VDI), screen/desktop sharing and collaboration, supervisory > control and data acquisition (SCADA) display, automotive/navigation > display, cloud gaming, factory automation display, wireless display, display > wall, digital operating room (DiOR), etc. > > What the difference between Section 2.1’s animation and Section 2.5’s > cartoons? > > What’s the different between Section 2.1’s “movies, TV series …” and Section > 2.2’s “television content”? > > (8) Section 2.5. The sentence “Currently, …” is very challenging to parse as it > includes inline “i.e.,” and “etc”. > > (9) Section 2.5. Per “powerpoint, word documents”, these are specific > Microsoft products. I recommend using more generic names. > > (10) Section 4. I found it confusing that an evaluation methodology was in a > requirements document. I would have expected it in the draft-ietf-netvc- > testing > > (11) Section 4.1. VP9 needs a reference. > > (12) Section 6. I don’t think this entire section is necessary. > > (13) Editorial Nits: > -- Section 3. Style nit. s/chapter/section/ > > -- Section 4.1. Typo. s/computged/computed >
- [video-codec] Roman Danyliw's Discuss on draft-ie… Roman Danyliw via Datatracker
- Re: [video-codec] Roman Danyliw's Discuss on draf… Adam Roach
- Re: [video-codec] Roman Danyliw's Discuss on draf… Filippov Alexey
- Re: [video-codec] Roman Danyliw's Discuss on draf… Roman Danyliw
- Re: [video-codec] Roman Danyliw's Discuss on draf… Filippov Alexey
- Re: [video-codec] Roman Danyliw's Discuss on draf… Roman Danyliw
- Re: [video-codec] Roman Danyliw's Discuss on draf… Filippov Alexey