Re: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)

[Roman Danyliw <[e-mail]>]

Hi Alexey!

> -----Original Message-----
> From: Filippov Alexey <[e-mail]>
> Sent: Tuesday, October 22, 2019 8:20 AM
> To: Roman Danyliw <[e-mail]>; Adam Roach <[e-mail]>; The
> IESG <[e-mail]>
> Cc: [e-mail]; [e-mail]; draft-ietf-netvc-
> [e-mail]; Mo Zanaty <[e-mail]>; Elena Alshina
> <[e-mail]>
> Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09:
> (with DISCUSS and COMMENT)
> 
> Dear Roman,
> 
> I apologize for a very late reply. Unfortunately, I really had no time to provide
> my feedback earlier. Please, find my comments below.
> 
> --
> Best regards,
> Alexey Filippov
> 
> >-----Original Message-----
> >From: Roman Danyliw [mailto:[e-mail]]
> >Sent: Friday, June 21, 2019 5:09 PM
> >To: Filippov Alexey <[e-mail]>; Adam Roach
> ><[e-mail]>; The IESG <[e-mail]>
> >Cc: [e-mail]; [e-mail];
> >[e-mail]; Mo Zanaty <[e-mail]>
> >Subject: RE: Roman Danyliw's Discuss on
> >draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)
> >
> >Hi!
> >
> >> -----Original Message-----
> >> From: iesg [mailto:[e-mail]] On Behalf Of Filippov
> >> Alexey
> >> Sent: Thursday, June 13, 2019 1:15 PM
> >> To: Roman Danyliw <[e-mail]>; Adam Roach <[e-mail]>;
> The
> >> IESG <[e-mail]>
> >> Cc: [e-mail]; [e-mail]; draft-ietf-netvc-
> >> [e-mail]; Mo Zanaty <[e-mail]>
> >> Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09:
> >> (with DISCUSS and COMMENT)
> >>
> >> Dear Roman,
> >>
> >> Thank you a lot for your comments and raising the important questions.
> >>
> >> > How should a set of QPs be specified?
> >> As you probably know, a set of available QP values can be codec specific.
> >> Usually, a QP set is selected to cover medium bit-rate range that is
> >> considered to be the most complex for compressing. We proposed to set
> >> up the assessment process not only for this range but also for low
> >> and high QP ranges to cover a wider range of applications. According
> >> to my understanding, concrete QP values should be specified for
> >> candidate codecs in draft-ietf-netvc-testing
> 
> >See below.  Can you help with a specific pointer into draft-ietf-netvc-testing
> which lists candidate codecs+QPs.
> AF: I guess the list of QPs for the candidate codec AV1 implemented in libaom
> is provided in Section 4.3 "Ranges" of draft-ietf-netvc-testing:
> "For the final evaluation described in [I-D.ietf-netvc-requirements], the
> quantizers used are 20, 24, 28, 32, 36, 39, 43, 47, 51, and 55. "

An explicit reference to netvc-testing would address my concern, and specifically make the following sentences clearer.

"Initially, for the codec selected as a reference one (e.g., HEVC or VP9), a set of 10 QP (quantization parameter) values should be specified (in a separate document on Internet video codec testing) and corresponding quality values should be calculated."

"A list of video sequences that should be used for testing as well as the 10 QP values for the reference codec are defined in a separate document."

Thanks.

> >>
> >> > How should the quality values be calculated?
> >> It is explained in Section 4.1, namely: “To assess the quality of
> >> output
> >> (decoded) sequences, two indexes, PSNR [3] and MS-SSIM [3,11] are
> >> separately computed. In the case of the YCbCr color format, PSNR
> >> should be calculated for each color plane whereas MS-SSIM is
> >> calculated for luma channel only. In the case of the RGB color
> >> format, both metrics are computed for R, G and B channels. Thus, for
> >> each sequence, 30 RD-points for PSNR (i.e. three RD-curves, one for
> >> each
> >> channel) and 10 RD-points for MS- SSIM (i.e. one RD-curve, for luma
> >> channel only) should be calculated in the case of YCbCr. If content
> >> is encoded as RGB, 60 RD-points (30 for PSNR and
> >> 30 for MS-SSIM) should be calculated, i.e. three RD-curves (one for
> >> each
> >> channel) are computed for PSNR as well as three RD-curves (one for
> >> each
> >> channel) for MS-SSIM.” In references [3, 11], these 2 quality
> >> assessment metrics and the ways of how to calculate them are described in
> detail.
> 
> >Got it.  Thanks.
> 
> >> >-- What does the text “(in a separate document on Internet video
> >> >codec
> >> testing)” mean?
> >> > Per “A list of video sequences that should be used for testing as
> >> > well as the
> >> 10 QP values for the reference codec are defined in a separate
> >> document ", what document is that?  Is it draft-ietf-netvc-testing?
> >> According to my understanding, it is draft-ietf-netvc-testing.
> >
> >Can you please help me with the specific section references in draft-ietf-netvc-
> testing as I'm not seeing it -- where is the explicit reference codec being named
> and its associate 10 QP values?  For what it's worth I see Section 4.1 of this
> draft saying that:
> 
> >   As
> >   the reference for evaluation, state-of-the-art video codecs such as
> >   HEVC/H.265 [4,5] or VP9  must be used. The reference source code of
> >   the HEVC/H.265 codec can be found at [6]. The HEVC/H.265 codec must
> >   be configured according to [13] and Table 9.
> >
> >I'm looking for something that either unambiguously points to the reference
> codec+QPs; or clearer language that says that all of this is out of scope in this
> and/or the -testing draft, but is clear on what input into the processes of this
> draft is required.
> AF: According to my understanding, VP9 implemented in libvpx is selected to
> be a reference codec and AV1 is a candidate (tested) one. So, the same QP
> range mentioned in Section 4.3 "Ranges" of draft-ietf-netvc-testing is
> applicable to both reference and candidate codecs.
> 
> >> >What does “codec implementation (for both an encoder and a decoder)
> >> should cover the worst case of computational complexity, memory
> >> bandwidth, and physical memory size” mean?
> >> I’d like to thank Adam for his comment. I also think that his
> >> formulation ("...should take into consideration the worst-case...") is clearer.
> >> A more detailed explanation on this Section can be found in my
> >> response to the secdir reviewer:
> >> https://mailarchive.ietf.org/arch/msg/secdir/QYx-
> >> hl07aec5XhgqkldqgWFG3Is
> >
> >I'd like to offer a further refinement of what Adam proposed:
> >
> >Original:
> >However, it is worth noting that a codec implementation (for both an encoder
> and a decoder) should cover the worst case of computational complexity,
> memory bandwidth, and physical memory size (e.g., for  decoded pictures used
> as references).
> >
> >Adam:
> >However, it is worth noting that a codec implementation (for both an encoder
> and a decoder) should take into consideration the worst-case computational
> complexity, memory bandwidth, and physical memory size (e.g., for decoded
> pictures used as references).
> >
> >Roman+Adam:
> >However, it is worth noting that a codec implementation (for both an encoder
> and a decoder) should take into consideration the worst-case computational
> complexity, memory bandwidth, and physical memory size needed to processes
> the input (e.g., the decoded pictures used as references).
> AF: The rephrasing labeled "Roman+Adam" sounds good to me. I guess it
> should replace the original sentence.

The proposed text above works for me.  Thanks.

> >> >Please add additional language that codec should be written in a
> >> >defensive
> >> style as they will be processing untrusted input.
> >> Good point. Thank you. I’ll add it.

Thanks.

Regards,
Roman

> >Regards,
> >Roman
> >
> >>
> >> --
> >> Best regards,
> >> Alexey Filippov
> >>
> >> -----Original Message-----
> >> From: Roman Danyliw via Datatracker [mailto:[e-mail]]
> >> Sent: Wednesday, June 12, 2019 4:25 AM
> >> To: The IESG <[e-mail]>
> >> Cc: [e-mail]; Mo Zanaty
> >> <[e-mail]>; [e-mail]; [e-mail]; video-
> >> [e-mail]
> >> Subject: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09:
> >> (with DISCUSS and COMMENT)
> >>
> >> Roman Danyliw has entered the following ballot position for
> >> draft-ietf-netvc-requirements-09: Discuss
> >>
> >> When responding, please keep the subject line intact and reply to all
> >> email addresses included in the To and CC lines. (Feel free to cut
> >> this introductory paragraph, however.)
> >>
> >>
> >> Please refer to
> >> https://www.ietf.org/iesg/statement/discuss-criteria.html
> >> for more information about IESG DISCUSS and COMMENT positions.
> >>
> >>
> >> The document, along with other ballot positions, can be found here:
> >> https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> -
> >> DISCUSS:
> >> ---------------------------------------------------------------------
> >> -
> >>
> >> (1) I worry that the level of detail in the in the Compression
> >> Performance Evaluation (Section 4.1) is insufficient for implementation.
> Specifically:
> >>
> >> (a) Per “Initially, for the codec selected as a reference one (e.g.,
> >> HEVC or VP9), a set of 10 QP quantization parameter) values should be
> >> specified (in a separate document on Internet video codec testing)
> >> and corresponding quality values should be calculated.”
> >>
> >> -- How should a set of QPs be specified?
> >>
> >> --How should the quality values be calculated?
> >>
> >> -- What does the text “(in a separate document on Internet video
> >> codec testing)” mean?
> >>
> >> (b) Per “A list of video sequences that should be used for testing as
> >> well as the 10 QP values for the reference codec are defined in a
> >> separate document ", what document is that?  Is it draft-ietf-netvc-testing?
> >>
> >> (2) Per the Security Considerations Section (Section 5)
> >>
> >> -- What does “codec implementation (for both an encoder and a
> >> decoder) should cover the worst case of computational complexity,
> >> memory bandwidth, and physical memory size” mean?
> >>
> >> -- Please add additional language that codec should be written in a
> >> defensive style as they will be processing untrusted input.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> -
> >> COMMENT:
> >> ---------------------------------------------------------------------
> >> -
> >>
> >> (1) There is something odd about the document formatting – the title
> >> and the first author last name in the footer is wrapped in “< … >”
> >>
> >> (2) It would be helpful to forward reference that acronyms are
> >> explained in Appendix A.
> >>
> >> (3) This draft uses the words should and must to prescribe action.
> >> Why wasn’t
> >> RFC2119 cited to explain these words?
> >>
> >> (4) Section 2.0.  A reference to explain “YCbCr 4:2:0” would be
> >> helpful
> >>
> >> (5) Section 2.1. Per “high encoder complexity” and “decoding
> >> complexity”, I initiate read that as a qualitative measure.  However,
> >> the text says “up to 10x and more” so that implies some quantitative
> measure.  What is that?
> >>
> >> (6) Section 2.1.  Expand QP values on first use
> >>
> >> (7) Section 2.x.  The language around content doesn’t appear to be consist.
> >> For example:
> >>
> >> -- Section 2.1, Internet Video Streaming says “movies, TV-series and
> >> shows, and animation.”
> >>
> >> -- Section 2.2, IPTV says “television content”
> >>
> >> -- Section 2.5, Screen casting says “business presentations …,
> >> animation (cartoons), gaming content, data visualization, …, virtual
> >> desktop infrastructure (VDI), screen/desktop sharing and
> >> collaboration, supervisory control and data acquisition (SCADA)
> >> display, automotive/navigation display, cloud gaming, factory
> >> automation display, wireless display, display wall, digital operating room
> (DiOR), etc.
> >>
> >> What the difference between Section 2.1’s animation and Section 2.5’s
> >> cartoons?
> >>
> >> What’s the different between Section 2.1’s “movies, TV series …” and
> >> Section 2.2’s “television content”?
> >>
> >> (8) Section 2.5.  The sentence “Currently, …” is very challenging to
> >> parse as it includes inline “i.e.,” and “etc”.
> >>
> >> (9) Section 2.5.  Per “powerpoint, word documents”, these are
> >> specific Microsoft products.  I recommend using more generic names.
> >>
> >> (10) Section 4.  I found it confusing that an evaluation methodology
> >> was in a requirements document.  I would have expected it in the
> >> draft-ietf-netvc- testing
> >>
> >> (11) Section 4.1.  VP9 needs a reference.
> >>
> >> (12) Section 6.  I don’t think this entire section is necessary.
> >>
> >> (13) Editorial Nits:
> >> -- Section 3.  Style nit.  s/chapter/section/
> >>
> >> -- Section 4.1. Typo.  s/computged/computed
> >>