Re: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)

[Filippov Alexey <[e-mail]>]

Dear Roman,

I apologize for a very late reply. Unfortunately, I really had no time to provide my feedback earlier. Please, find my comments below.

--
Best regards,
Alexey Filippov

>-----Original Message-----
>From: Roman Danyliw [mailto:[e-mail]] 
>Sent: Friday, June 21, 2019 5:09 PM
>To: Filippov Alexey <[e-mail]>; Adam Roach <[e-mail]>; The IESG <[e-mail]>
>Cc: [e-mail]; [e-mail]; [e-mail]; Mo Zanaty <[e-mail]>
>Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)
>
>Hi!
>
>> -----Original Message-----
>> From: iesg [mailto:[e-mail]] On Behalf Of Filippov Alexey
>> Sent: Thursday, June 13, 2019 1:15 PM
>> To: Roman Danyliw <[e-mail]>; Adam Roach <[e-mail]>; The 
>> IESG <[e-mail]>
>> Cc: [e-mail]; [e-mail]; draft-ietf-netvc- 
>> [e-mail]; Mo Zanaty <[e-mail]>
>> Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09:
>> (with DISCUSS and COMMENT)
>> 
>> Dear Roman,
>> 
>> Thank you a lot for your comments and raising the important questions.
>> 
>> > How should a set of QPs be specified?
>> As you probably know, a set of available QP values can be codec specific.
>> Usually, a QP set is selected to cover medium bit-rate range that is 
>> considered to be the most complex for compressing. We proposed to set 
>> up the assessment process not only for this range but also for low and 
>> high QP ranges to cover a wider range of applications. According to my 
>> understanding, concrete QP values should be specified for candidate 
>> codecs in draft-ietf-netvc-testing

>See below.  Can you help with a specific pointer into draft-ietf-netvc-testing which lists candidate codecs+QPs.
AF: I guess the list of QPs for the candidate codec AV1 implemented in libaom is provided in Section 4.3 "Ranges" of draft-ietf-netvc-testing:
"For the final evaluation described in [I-D.ietf-netvc-requirements], the quantizers used are 20, 24, 28, 32, 36, 39, 43, 47, 51, and 55. " 

>> 
>> > How should the quality values be calculated?
>> It is explained in Section 4.1, namely: “To assess the quality of 
>> output
>> (decoded) sequences, two indexes, PSNR [3] and MS-SSIM [3,11] are 
>> separately computed. In the case of the YCbCr color format, PSNR 
>> should be calculated for each color plane whereas MS-SSIM is 
>> calculated for luma channel only. In the case of the RGB color format, 
>> both metrics are computed for R, G and B channels. Thus, for each 
>> sequence, 30 RD-points for PSNR (i.e. three RD-curves, one for each 
>> channel) and 10 RD-points for MS- SSIM (i.e. one RD-curve, for luma 
>> channel only) should be calculated in the case of YCbCr. If content is 
>> encoded as RGB, 60 RD-points (30 for PSNR and
>> 30 for MS-SSIM) should be calculated, i.e. three RD-curves (one for 
>> each
>> channel) are computed for PSNR as well as three RD-curves (one for 
>> each
>> channel) for MS-SSIM.” In references [3, 11], these 2 quality 
>> assessment metrics and the ways of how to calculate them are described in detail.

>Got it.  Thanks.

>> >-- What does the text “(in a separate document on Internet video 
>> >codec
>> testing)” mean?
>> > Per “A list of video sequences that should be used for testing as 
>> > well as the
>> 10 QP values for the reference codec are defined in a separate 
>> document ", what document is that?  Is it draft-ietf-netvc-testing?
>> According to my understanding, it is draft-ietf-netvc-testing.
>
>Can you please help me with the specific section references in draft-ietf-netvc-testing as I'm not seeing it -- where is the explicit reference codec being named and its associate 10 QP values?  For what it's worth I see Section 4.1 of this draft saying that:

>   As
>   the reference for evaluation, state-of-the-art video codecs such as
>   HEVC/H.265 [4,5] or VP9  must be used. The reference source code of
>   the HEVC/H.265 codec can be found at [6]. The HEVC/H.265 codec must
>   be configured according to [13] and Table 9.
>
>I'm looking for something that either unambiguously points to the reference codec+QPs; or clearer language that says that all of this is out of scope in this and/or the -testing draft, but is clear on what input into the processes of this draft is required.
AF: According to my understanding, VP9 implemented in libvpx is selected to be a reference codec and AV1 is a candidate (tested) one. So, the same QP range mentioned in Section 4.3 "Ranges" of draft-ietf-netvc-testing is applicable to both reference and candidate codecs.

>> >What does “codec implementation (for both an encoder and a decoder)
>> should cover the worst case of computational complexity, memory 
>> bandwidth, and physical memory size” mean?
>> I’d like to thank Adam for his comment. I also think that his 
>> formulation ("...should take into consideration the worst-case...") is clearer.
>> A more detailed explanation on this Section can be found in my 
>> response to the secdir reviewer: 
>> https://mailarchive.ietf.org/arch/msg/secdir/QYx-
>> hl07aec5XhgqkldqgWFG3Is
>
>I'd like to offer a further refinement of what Adam proposed:
>
>Original:
>However, it is worth noting that a codec implementation (for both an encoder and a decoder) should cover the worst case of computational complexity, memory bandwidth, and physical memory size (e.g., for  decoded pictures used as references).
>
>Adam:
>However, it is worth noting that a codec implementation (for both an encoder and a decoder) should take into consideration the worst-case computational complexity, memory bandwidth, and physical memory size (e.g., for decoded pictures used as references).
>
>Roman+Adam:
>However, it is worth noting that a codec implementation (for both an encoder and a decoder) should take into consideration the worst-case computational complexity, memory bandwidth, and physical memory size needed to processes the input (e.g., the decoded pictures used as references).
AF: The rephrasing labeled "Roman+Adam" sounds good to me. I guess it should replace the original sentence.


>> >Please add additional language that codec should be written in a 
>> >defensive
>> style as they will be processing untrusted input.
>> Good point. Thank you. I’ll add it.
>
>Regards,
>Roman
>
>> 
>> --
>> Best regards,
>> Alexey Filippov
>> 
>> -----Original Message-----
>> From: Roman Danyliw via Datatracker [mailto:[e-mail]]
>> Sent: Wednesday, June 12, 2019 4:25 AM
>> To: The IESG <[e-mail]>
>> Cc: [e-mail]; Mo Zanaty 
>> <[e-mail]>; [e-mail]; [e-mail]; video- 
>> [e-mail]
>> Subject: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: 
>> (with DISCUSS and COMMENT)
>> 
>> Roman Danyliw has entered the following ballot position for
>> draft-ietf-netvc-requirements-09: Discuss
>> 
>> When responding, please keep the subject line intact and reply to all 
>> email addresses included in the To and CC lines. (Feel free to cut 
>> this introductory paragraph, however.)
>> 
>> 
>> Please refer to 
>> https://www.ietf.org/iesg/statement/discuss-criteria.html
>> for more information about IESG DISCUSS and COMMENT positions.
>> 
>> 
>> The document, along with other ballot positions, can be found here:
>> https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/
>> 
>> 
>> 
>> ----------------------------------------------------------------------
>> DISCUSS:
>> ----------------------------------------------------------------------
>> 
>> (1) I worry that the level of detail in the in the Compression 
>> Performance Evaluation (Section 4.1) is insufficient for implementation.  Specifically:
>> 
>> (a) Per “Initially, for the codec selected as a reference one (e.g., 
>> HEVC or VP9), a set of 10 QP quantization parameter) values should be 
>> specified (in a separate document on Internet video codec testing) and 
>> corresponding quality values should be calculated.”
>> 
>> -- How should a set of QPs be specified?
>> 
>> --How should the quality values be calculated?
>> 
>> -- What does the text “(in a separate document on Internet video codec 
>> testing)” mean?
>> 
>> (b) Per “A list of video sequences that should be used for testing as 
>> well as the 10 QP values for the reference codec are defined in a 
>> separate document ", what document is that?  Is it draft-ietf-netvc-testing?
>> 
>> (2) Per the Security Considerations Section (Section 5)
>> 
>> -- What does “codec implementation (for both an encoder and a decoder) 
>> should cover the worst case of computational complexity, memory 
>> bandwidth, and physical memory size” mean?
>> 
>> -- Please add additional language that codec should be written in a 
>> defensive style as they will be processing untrusted input.
>> 
>> 
>> ----------------------------------------------------------------------
>> COMMENT:
>> ----------------------------------------------------------------------
>> 
>> (1) There is something odd about the document formatting – the title 
>> and the first author last name in the footer is wrapped in “< … >”
>> 
>> (2) It would be helpful to forward reference that acronyms are 
>> explained in Appendix A.
>> 
>> (3) This draft uses the words should and must to prescribe action.  
>> Why wasn’t
>> RFC2119 cited to explain these words?
>> 
>> (4) Section 2.0.  A reference to explain “YCbCr 4:2:0” would be 
>> helpful
>> 
>> (5) Section 2.1. Per “high encoder complexity” and “decoding 
>> complexity”, I initiate read that as a qualitative measure.  However, 
>> the text says “up to 10x and more” so that implies some quantitative measure.  What is that?
>> 
>> (6) Section 2.1.  Expand QP values on first use
>> 
>> (7) Section 2.x.  The language around content doesn’t appear to be consist.
>> For example:
>> 
>> -- Section 2.1, Internet Video Streaming says “movies, TV-series and 
>> shows, and animation.”
>> 
>> -- Section 2.2, IPTV says “television content”
>> 
>> -- Section 2.5, Screen casting says “business presentations …, 
>> animation (cartoons), gaming content, data visualization, …, virtual 
>> desktop infrastructure (VDI), screen/desktop sharing and 
>> collaboration, supervisory control and data acquisition (SCADA) 
>> display, automotive/navigation display, cloud gaming, factory 
>> automation display, wireless display, display wall, digital operating room (DiOR), etc.
>> 
>> What the difference between Section 2.1’s animation and Section 2.5’s 
>> cartoons?
>> 
>> What’s the different between Section 2.1’s “movies, TV series …” and 
>> Section 2.2’s “television content”?
>> 
>> (8) Section 2.5.  The sentence “Currently, …” is very challenging to 
>> parse as it includes inline “i.e.,” and “etc”.
>> 
>> (9) Section 2.5.  Per “powerpoint, word documents”, these are specific 
>> Microsoft products.  I recommend using more generic names.
>> 
>> (10) Section 4.  I found it confusing that an evaluation methodology 
>> was in a requirements document.  I would have expected it in the 
>> draft-ietf-netvc- testing
>> 
>> (11) Section 4.1.  VP9 needs a reference.
>> 
>> (12) Section 6.  I don’t think this entire section is necessary.
>> 
>> (13) Editorial Nits:
>> -- Section 3.  Style nit.  s/chapter/section/
>> 
>> -- Section 4.1. Typo.  s/computged/computed
>>