Re: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)

[Filippov Alexey <[e-mail]>]

Dear Roman,

Thank you a lot for your comments and raising the important questions.

> How should a set of QPs be specified?
As you probably know, a set of available QP values can be codec specific. Usually, a QP set is selected to cover medium bit-rate range that is considered to be the most complex for compressing. We proposed to set up the assessment process not only for this range but also for low and high QP ranges to cover a wider range of applications. According to my understanding, concrete QP values should be specified for candidate codecs in draft-ietf-netvc-testing.

> How should the quality values be calculated?
It is explained in Section 4.1, namely: “To assess the quality of output (decoded) sequences, two indexes, PSNR [3] and MS-SSIM [3,11] are separately computed. In the case of the YCbCr color format, PSNR should be calculated for each color plane whereas MS-SSIM is calculated for luma channel only. In the case of the RGB color format, both metrics are computed for R, G and B channels. Thus, for each sequence, 30 RD-points for PSNR (i.e. three RD-curves, one for each channel) and 10 RD-points for MS-SSIM (i.e. one RD-curve, for luma channel only) should be calculated in the case of YCbCr. If content is encoded as RGB, 60 RD-points (30 for PSNR and 30 for MS-SSIM) should be calculated, i.e. three RD-curves (one for each channel) are computed for PSNR as well as three RD-curves (one for each channel) for MS-SSIM.” In references [3, 11], these 2 quality assessment metrics and the ways of how to calculate them are described in detail.

>-- What does the text “(in a separate document on Internet video codec testing)” mean?
> Per “A list of video sequences that should be used for testing as well as the 10 QP values for the reference codec are defined in a separate document ", what document is that?  Is it draft-ietf-netvc-testing?
According to my understanding, it is draft-ietf-netvc-testing.

>What does “codec implementation (for both an encoder and a decoder) should cover the worst case of computational complexity, memory bandwidth, and physical memory size” mean?
I’d like to thank Adam for his comment. I also think that his formulation ("...should take into consideration the worst-case...") is clearer. 
A more detailed explanation on this Section can be found in my response to the secdir reviewer: https://mailarchive.ietf.org/arch/msg/secdir/QYx-hl07aec5XhgqkldqgWFG3Is 

>Please add additional language that codec should be written in a defensive style as they will be processing untrusted input.
Good point. Thank you. I’ll add it.

--
Best regards,
Alexey Filippov

-----Original Message-----
From: Roman Danyliw via Datatracker [mailto:[e-mail]] 
Sent: Wednesday, June 12, 2019 4:25 AM
To: The IESG <[e-mail]>
Cc: [e-mail]; Mo Zanaty <[e-mail]>; [e-mail]; [e-mail]; [e-mail]
Subject: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)

Roman Danyliw has entered the following ballot position for
draft-ietf-netvc-requirements-09: Discuss

When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.)


Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.


The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/



----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

(1) I worry that the level of detail in the in the Compression Performance Evaluation (Section 4.1) is insufficient for implementation.  Specifically:

(a) Per “Initially, for the codec selected as a reference one (e.g., HEVC or VP9), a set of 10 QP quantization parameter) values should be specified (in a separate document on Internet video codec testing) and corresponding quality values should be calculated.”

-- How should a set of QPs be specified?

--How should the quality values be calculated?

-- What does the text “(in a separate document on Internet video codec testing)” mean?

(b) Per “A list of video sequences that should be used for testing as well as the 10 QP values for the reference codec are defined in a separate document ", what document is that?  Is it draft-ietf-netvc-testing?

(2) Per the Security Considerations Section (Section 5)

-- What does “codec implementation (for both an encoder and a decoder) should cover the worst case of computational complexity, memory bandwidth, and physical memory size” mean?

-- Please add additional language that codec should be written in a defensive style as they will be processing untrusted input.


----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

(1) There is something odd about the document formatting – the title and the first author last name in the footer is wrapped in “< … >”

(2) It would be helpful to forward reference that acronyms are explained in Appendix A.

(3) This draft uses the words should and must to prescribe action.  Why wasn’t
RFC2119 cited to explain these words?

(4) Section 2.0.  A reference to explain “YCbCr 4:2:0” would be helpful

(5) Section 2.1. Per “high encoder complexity” and “decoding complexity”, I initiate read that as a qualitative measure.  However, the text says “up to 10x and more” so that implies some quantitative measure.  What is that?

(6) Section 2.1.  Expand QP values on first use

(7) Section 2.x.  The language around content doesn’t appear to be consist. 
For example:

-- Section 2.1, Internet Video Streaming says “movies, TV-series and shows, and animation.”

-- Section 2.2, IPTV says “television content”

-- Section 2.5, Screen casting says “business presentations …, animation (cartoons), gaming content, data visualization, …, virtual desktop infrastructure (VDI), screen/desktop sharing and collaboration, supervisory control and data acquisition (SCADA) display, automotive/navigation display, cloud gaming, factory automation display, wireless display, display wall, digital operating room (DiOR), etc.

What the difference between Section 2.1’s animation and Section 2.5’s cartoons?

What’s the different between Section 2.1’s “movies, TV series …” and Section 2.2’s “television content”?

(8) Section 2.5.  The sentence “Currently, …” is very challenging to parse as it includes inline “i.e.,” and “etc”.

(9) Section 2.5.  Per “powerpoint, word documents”, these are specific Microsoft products.  I recommend using more generic names.

(10) Section 4.  I found it confusing that an evaluation methodology was in a requirements document.  I would have expected it in the draft-ietf-netvc-testing

(11) Section 4.1.  VP9 needs a reference.

(12) Section 6.  I don’t think this entire section is necessary.

(13) Editorial Nits:
-- Section 3.  Style nit.  s/chapter/section/

-- Section 4.1. Typo.  s/computged/computed