Re: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)

[Filippov Alexey <[e-mail]>]

Hi Roman,

-----Original Message-----
From: Roman Danyliw [mailto:[e-mail]] 
Sent: Friday, November 15, 2019 10:01 AM
To: Filippov Alexey <[e-mail]>; Adam Roach <[e-mail]>; The IESG <[e-mail]>
Cc: [e-mail]; [e-mail]; [e-mail]; Mo Zanaty <[e-mail]>; Elena Alshina <[e-mail]>
Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)

>Hi Alexey!

>> -----Original Message-----
>> From: Filippov Alexey <[e-mail]>
>> Sent: Tuesday, October 22, 2019 8:20 AM
>> To: Roman Danyliw <[e-mail]>; Adam Roach <[e-mail]>; The 
>> IESG <[e-mail]>
>> Cc: [e-mail]; [e-mail]; draft-ietf-netvc- 
>> [e-mail]; Mo Zanaty <[e-mail]>; Elena Alshina 
>> <[e-mail]>
>> Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09:
>> (with DISCUSS and COMMENT)
>> 
>> Dear Roman,
>> 
>> I apologize for a very late reply. Unfortunately, I really had no time 
>> to provide my feedback earlier. Please, find my comments below.
>> 
>> --
>> Best regards,
>> Alexey Filippov
>> 
>> >-----Original Message-----
>> >From: Roman Danyliw [mailto:[e-mail]]
>> >Sent: Friday, June 21, 2019 5:09 PM
>> >To: Filippov Alexey <[e-mail]>; Adam Roach 
>> ><[e-mail]>; The IESG <[e-mail]>
>> >Cc: [e-mail]; [e-mail]; 
>> >[e-mail]; Mo Zanaty <[e-mail]>
>> >Subject: RE: Roman Danyliw's Discuss on
>> >draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)
>> >
>> >Hi!
>> >
>> >> -----Original Message-----
>> >> From: iesg [mailto:[e-mail]] On Behalf Of Filippov 
>> >> Alexey
>> >> Sent: Thursday, June 13, 2019 1:15 PM
>> >> To: Roman Danyliw <[e-mail]>; Adam Roach <[e-mail]>;
>> The
>> >> IESG <[e-mail]>
>> >> Cc: [e-mail]; [e-mail]; draft-ietf-netvc- 
>> >> [e-mail]; Mo Zanaty <[e-mail]>
>> >> Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09:
>> >> (with DISCUSS and COMMENT)
>> >>
>> >> Dear Roman,
>> >>
>> >> Thank you a lot for your comments and raising the important questions.
>> >>
>> >> > How should a set of QPs be specified?
>> >> As you probably know, a set of available QP values can be codec specific.
>> >> Usually, a QP set is selected to cover medium bit-rate range that 
>> >> is considered to be the most complex for compressing. We proposed 
>> >> to set up the assessment process not only for this range but also 
>> >> for low and high QP ranges to cover a wider range of applications. 
>> >> According to my understanding, concrete QP values should be 
>> >> specified for candidate codecs in draft-ietf-netvc-testing
>> 
>> >See below.  Can you help with a specific pointer into 
>> >draft-ietf-netvc-testing
>> which lists candidate codecs+QPs.
>> AF: I guess the list of QPs for the candidate codec AV1 implemented in 
>> libaom is provided in Section 4.3 "Ranges" of draft-ietf-netvc-testing:
>> "For the final evaluation described in [I-D.ietf-netvc-requirements], 
>> the quantizers used are 20, 24, 28, 32, 36, 39, 43, 47, 51, and 55. "
>
>>An explicit reference to netvc-testing would address my concern, and specifically make the following sentences clearer. >
>
>"Initially, for the codec selected as a reference one (e.g., HEVC or VP9), a set of 10 QP (quantization parameter) values should be specified (in a separate document on Internet video codec testing) and corresponding quality values should be calculated."
>
>"A list of video sequences that should be used for testing as well as the 10 QP values for the reference codec are defined in a separate document."
>
>Thanks.
AF: Thank you for your comments. Explicit references to the -testing draft will be added to the next revision of the document.

>> >>
>> >> > How should the quality values be calculated?
>> >> It is explained in Section 4.1, namely: “To assess the quality of 
>> >> output
>> >> (decoded) sequences, two indexes, PSNR [3] and MS-SSIM [3,11] are 
>> >> separately computed. In the case of the YCbCr color format, PSNR 
>> >> should be calculated for each color plane whereas MS-SSIM is 
>> >> calculated for luma channel only. In the case of the RGB color 
>> >> format, both metrics are computed for R, G and B channels. Thus, 
>> >> for each sequence, 30 RD-points for PSNR (i.e. three RD-curves, one 
>> >> for each
>> >> channel) and 10 RD-points for MS- SSIM (i.e. one RD-curve, for luma 
>> >> channel only) should be calculated in the case of YCbCr. If content 
>> >> is encoded as RGB, 60 RD-points (30 for PSNR and
>> >> 30 for MS-SSIM) should be calculated, i.e. three RD-curves (one for 
>> >> each
>> >> channel) are computed for PSNR as well as three RD-curves (one for 
>> >> each
>> >> channel) for MS-SSIM.” In references [3, 11], these 2 quality 
>> >> assessment metrics and the ways of how to calculate them are 
>> >> described in
>> detail.
>> 
>> >Got it.  Thanks.
>> 
>> >> >-- What does the text “(in a separate document on Internet video 
>> >> >codec
>> >> testing)” mean?
>> >> > Per “A list of video sequences that should be used for testing as 
>> >> > well as the
>> >> 10 QP values for the reference codec are defined in a separate 
>> >> document ", what document is that?  Is it draft-ietf-netvc-testing?
>> >> According to my understanding, it is draft-ietf-netvc-testing.
>> >
>> >Can you please help me with the specific section references in 
>> >draft-ietf-netvc-
>> testing as I'm not seeing it -- where is the explicit reference codec 
>> being named and its associate 10 QP values?  For what it's worth I see 
>> Section 4.1 of this draft saying that:
>> 
>> >   As
>> >   the reference for evaluation, state-of-the-art video codecs such as
>> >   HEVC/H.265 [4,5] or VP9  must be used. The reference source code of
>> >   the HEVC/H.265 codec can be found at [6]. The HEVC/H.265 codec must
>> >   be configured according to [13] and Table 9.
>> >
>> >I'm looking for something that either unambiguously points to the 
>> >reference
>> codec+QPs; or clearer language that says that all of this is out of 
>> codec+scope in this
>> and/or the -testing draft, but is clear on what input into the 
>> processes of this draft is required.
>> AF: According to my understanding, VP9 implemented in libvpx is 
>> selected to be a reference codec and AV1 is a candidate (tested) one. 
>> So, the same QP range mentioned in Section 4.3 "Ranges" of 
>> draft-ietf-netvc-testing is applicable to both reference and candidate codecs.
>> 
>> >> >What does “codec implementation (for both an encoder and a 
>> >> >decoder)
>> >> should cover the worst case of computational complexity, memory 
>> >> bandwidth, and physical memory size” mean?
>> >> I’d like to thank Adam for his comment. I also think that his 
>> >> formulation ("...should take into consideration the worst-case...") is clearer.
>> >> A more detailed explanation on this Section can be found in my 
>> >> response to the secdir reviewer:
>> >> https://mailarchive.ietf.org/arch/msg/secdir/QYx-
>> >> hl07aec5XhgqkldqgWFG3Is
>> >
>> >I'd like to offer a further refinement of what Adam proposed:
>> >
>> >Original:
>> >However, it is worth noting that a codec implementation (for both an 
>> >encoder
>> and a decoder) should cover the worst case of computational 
>> complexity, memory bandwidth, and physical memory size (e.g., for  
>> decoded pictures used as references).
>> >
>> >Adam:
>> >However, it is worth noting that a codec implementation (for both an 
>> >encoder
>> and a decoder) should take into consideration the worst-case 
>> computational complexity, memory bandwidth, and physical memory size 
>> (e.g., for decoded pictures used as references).
>> >
>> >Roman+Adam:
>> >However, it is worth noting that a codec implementation (for both an 
>> >encoder
>> and a decoder) should take into consideration the worst-case 
>> computational complexity, memory bandwidth, and physical memory size 
>> needed to processes the input (e.g., the decoded pictures used as references).
>> AF: The rephrasing labeled "Roman+Adam" sounds good to me. I guess it 
>> should replace the original sentence.
>
>The proposed text above works for me.  Thanks.
AF: Good to know. Thank you!

--
Best regards,
Alexey Filippov

>> >> >Please add additional language that codec should be written in a 
>> >> >defensive
>> >> style as they will be processing untrusted input.
>> >> Good point. Thank you. I’ll add it.
>
>Thanks.
>
>Regards,
>Roman
>
>> >Regards,
>> >Roman
>> >
>> >>
>> >> --
>> >> Best regards,
>> >> Alexey Filippov
>> >>
>> >> -----Original Message-----
>> >> From: Roman Danyliw via Datatracker [mailto:[e-mail]]
>> >> Sent: Wednesday, June 12, 2019 4:25 AM
>> >> To: The IESG <[e-mail]>
>> >> Cc: [e-mail]; Mo Zanaty 
>> >> <[e-mail]>; [e-mail]; [e-mail]; 
>> >> video- [e-mail]
>> >> Subject: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09:
>> >> (with DISCUSS and COMMENT)
>> >>
>> >> Roman Danyliw has entered the following ballot position for
>> >> draft-ietf-netvc-requirements-09: Discuss
>> >>
>> >> When responding, please keep the subject line intact and reply to 
>> >> all email addresses included in the To and CC lines. (Feel free to 
>> >> cut this introductory paragraph, however.)
>> >>
>> >>
>> >> Please refer to
>> >> https://www.ietf.org/iesg/statement/discuss-criteria.html
>> >> for more information about IESG DISCUSS and COMMENT positions.
>> >>
>> >>
>> >> The document, along with other ballot positions, can be found here:
>> >> https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/
>> >>
>> >>
>> >>
>> >> -------------------------------------------------------------------
>> >> --
>> >> -
>> >> DISCUSS:
>> >> -------------------------------------------------------------------
>> >> --
>> >> -
>> >>
>> >> (1) I worry that the level of detail in the in the Compression 
>> >> Performance Evaluation (Section 4.1) is insufficient for implementation.
>> Specifically:
>> >>
>> >> (a) Per “Initially, for the codec selected as a reference one 
>> >> (e.g., HEVC or VP9), a set of 10 QP quantization parameter) values 
>> >> should be specified (in a separate document on Internet video codec 
>> >> testing) and corresponding quality values should be calculated.”
>> >>
>> >> -- How should a set of QPs be specified?
>> >>
>> >> --How should the quality values be calculated?
>> >>
>> >> -- What does the text “(in a separate document on Internet video 
>> >> codec testing)” mean?
>> >>
>> >> (b) Per “A list of video sequences that should be used for testing 
>> >> as well as the 10 QP values for the reference codec are defined in 
>> >> a separate document ", what document is that?  Is it draft-ietf-netvc-testing?
>> >>
>> >> (2) Per the Security Considerations Section (Section 5)
>> >>
>> >> -- What does “codec implementation (for both an encoder and a
>> >> decoder) should cover the worst case of computational complexity, 
>> >> memory bandwidth, and physical memory size” mean?
>> >>
>> >> -- Please add additional language that codec should be written in a 
>> >> defensive style as they will be processing untrusted input.
>> >>
>> >>
>> >> -------------------------------------------------------------------
>> >> --
>> >> -
>> >> COMMENT:
>> >> -------------------------------------------------------------------
>> >> --
>> >> -
>> >>
>> >> (1) There is something odd about the document formatting – the 
>> >> title and the first author last name in the footer is wrapped in “< … >”
>> >>
>> >> (2) It would be helpful to forward reference that acronyms are 
>> >> explained in Appendix A.
>> >>
>> >> (3) This draft uses the words should and must to prescribe action.
>> >> Why wasn’t
>> >> RFC2119 cited to explain these words?
>> >>
>> >> (4) Section 2.0.  A reference to explain “YCbCr 4:2:0” would be 
>> >> helpful
>> >>
>> >> (5) Section 2.1. Per “high encoder complexity” and “decoding 
>> >> complexity”, I initiate read that as a qualitative measure.  
>> >> However, the text says “up to 10x and more” so that implies some 
>> >> quantitative
>> measure.  What is that?
>> >>
>> >> (6) Section 2.1.  Expand QP values on first use
>> >>
>> >> (7) Section 2.x.  The language around content doesn’t appear to be consist.
>> >> For example:
>> >>
>> >> -- Section 2.1, Internet Video Streaming says “movies, TV-series 
>> >> and shows, and animation.”
>> >>
>> >> -- Section 2.2, IPTV says “television content”
>> >>
>> >> -- Section 2.5, Screen casting says “business presentations …, 
>> >> animation (cartoons), gaming content, data visualization, …, 
>> >> virtual desktop infrastructure (VDI), screen/desktop sharing and 
>> >> collaboration, supervisory control and data acquisition (SCADA) 
>> >> display, automotive/navigation display, cloud gaming, factory 
>> >> automation display, wireless display, display wall, digital 
>>> >> operating room
>> (DiOR), etc.
>> >>
>> >> What the difference between Section 2.1’s animation and Section 
>>>> >> 2.5’s cartoons?
>> >>
>> >> What’s the different between Section 2.1’s “movies, TV series …” 
>> >> and Section 2.2’s “television content”?
>> >>
>> >> (8) Section 2.5.  The sentence “Currently, …” is very challenging 
>> >> to parse as it includes inline “i.e.,” and “etc”.
>> >>
>> >> (9) Section 2.5.  Per “powerpoint, word documents”, these are 
>> >> specific Microsoft products.  I recommend using more generic names.
>> >>
>> >> (10) Section 4.  I found it confusing that an evaluation 
>> >> methodology was in a requirements document.  I would have expected 
>>> >> it in the
>> >> draft-ietf-netvc- testing
> >>>
>> >> (11) Section 4.1.  VP9 needs a reference.
> >>>
> >>> (12) Section 6.  I don’t think this entire section is necessary.
> >>>
> >>> (13) Editorial Nits:
> >>> -- Section 3.  Style nit.  s/chapter/section/
> >>>
> >>> -- Section 4.1. Typo.  s/computged/computed
> >>>